Working with Python

System Commands

To embrace the unix way, letting each tool do it’s thing, you need to be able to execute a system command from Python. There are a few ways to call shell commands, but the modern and recommended way is with the subprocess module.

`os.system()`

The older and very basic way was to use os.system() which is good for “one-liner” tasks that you don’t care about output or errors, just want it to run.

import os
exit_code = os.system('ls -l')
print("Exit code:", exit_code)

An exit code of 0 typically indicates success.

Downsides: You get only the exit code, not the output itself. For anything more robust, use subprocess.

`subprocess.run()`

An easy and more flexible way to run a command and get results is to use the subprocess modules. The subprocess.run() function takes a list of command arguments as its first parameter, with the command and each argument as a separate string.

Basic Example

import subprocess
subprocess.run(["ls", "-l"])
# prints directory listing

Capture Command Output

The above command runs and displays the output directly, as-if you ran it yourself. If you want to capture that output to a string. Use the capture_output=True parameter. The text=True parameter is also important here, as it decodes the command’s output from bytes to a string using the default system encoding.

result = subprocess.run(["ls", "-l"], capture_output=True, text=True)
print(result.stdout)  # output as a string

Handling Timeouts

Sometimes, an external command might hang indefinitely. To prevent your Python script from also hanging, you can use the timeout parameter (in seconds). If the command exceeds this timeout, a subprocess.TimeoutExpired exception is raised.

import subprocess
try:
    result = subprocess.run(["sleep", "10"], timeout=5, capture_output=True, text=True)
    print("Command finished within timeout")
except subprocess.TimeoutExpired:
    print("Command timed out!")
except subprocess.CalledProcessError as e:
    print(f"Command failed: {e}")

Understanding Encodings

When text=True, subprocess uses the system’s default encoding to decode output from commands. Usually, this is UTF-8 and works fine. However, if a command outputs text in a different encoding, you might see garbled characters or errors. In such cases, you can specify the encoding explicitly:

# Example assuming a command outputs Latin-1 encoded text
result = subprocess.run(["some_command_with_latin1_output"], capture_output=True, text=True, encoding='latin-1')
print(result.stdout)

If you are dealing with binary data, or need precise control over byte streams, you should omit text=True (and encoding) and work directly with the result.stdout and result.stderr as byte strings.

Run a command and check for errors

Commands fail sometimes, catch the error.

try:
    subprocess.run(["git", "status"], check=True)
except subprocess.CalledProcessError as e:
    print(f"The git command failed with exit code {e.returncode}")
    if e.stdout:
        print(f"Stdout:\n{e.stdout}")
    if e.stderr:
        print(f"Stderr:\n{e.stderr}")

Running with a shell

What does it mean to run with a shell?

When you use shell=True, Python runs the command through your shell (like bash or zsh), letting you:

Use pipes: cat foo.txt | grep bar
Expand wildcards (*.py)
Use environment variables ($HOME)

Example:

subprocess.run("echo $HOME", shell=True)
# prints your home directory

Security Warning: Using shell=True with untrusted input can be a security risk due to potential shell injection vulnerabilities. If any part of the command comes from external input (e.g., user input, web request), it’s highly recommended to avoid shell=True or to meticulously sanitize the input. Prefer passing arguments as a sequence (e.g., ["ls", "-l"]) whenever possible, as this is inherently safer.

Advanced: `subprocess.Popen`

Popen lets you do advanced things:

Read output as it is produced (line by line)
Send input to a command while it runs
Pipe output between commands (like chaining with |)
Start background processes

Example: Stream output as command runs

import subprocess
proc = subprocess.Popen(["ping", "-c", "3", "example.com"], stdout=subprocess.PIPE, text=True)
for line in proc.stdout:
    print("got:", line.rstrip())

Example: Send input to a process

proc = subprocess.Popen(["cat"], stdin=subprocess.PIPE, text=True)
proc.communicate(input="Hello from Python!\n")
# 'cat' will echo this right back out

Example: Pipe output between commands

from subprocess import Popen, PIPE
ps = Popen(["ps", "aux"], stdout=PIPE)
grep = Popen(["grep", "python"], stdin=ps.stdout, stdout=PIPE, text=True)
ps.stdout.close()  # Allow ps to get SIGPIPE if grep closes
output = grep.communicate()[0]
print(output)

For most scripts, you’ll only need subprocess.run(), which can also accept input via its input argument (e.g., subprocess.run(["grep", "pattern"], input="some text", text=True)). You only need to use Popen for advanced features.