Calling Shell Commands from Python: OS.system vs Subprocess

If you are a Python programmer, it is quite likely that you have experience in shell scripting. It is not uncommon to face a task that seems trivial to solve with a shell command. Therefore, it is useful to be familiar with how to call these commands efficiently from your Python code and know their limitations.

In this short article, I discuss how to use the older (although still relatively common)

os.system

command and the newer

subprocess

command. I will show some of their potential risks, limitations and provide complete examples of their use.

External commands might be an attractive option for small tasks if you are familiar with them and the pythonic alternative would imply more learning/implementation time. However, Python functions will save you time and trouble in the long run.

Option 1: OS.system (deprecated)

I show below a simple example of

os.system

to print the first line of a file provided by the user, using

head -n 1

#!/usr/bin/env python3
import os
# Define command and options wanted
command = "head -n 1 "
# Ask user for file name(s) - SECURITY RISK: susceptible to shell injection
filename = input("Please introduce name of file of interest:\n")
# Run os.system and save return_value
return_value = os.system(command+filename)
print('###############')
print('Return Value:', return_value)

The

os.system

function is easy to use and interpret: simply use as input of the function the same command you would use in a shell. However, it is deprecated and it is recommended to use

subprocess

now.

Note that this function will simply execute the shell command and the result will be printed to the standard output, but the output that the function returns is the return value (0 if it ran OK, and different than 0 otherwise).

1.1: Susceptibility to shell injection

Among other drawbacks,

os.system

directly executes the command in a shell, which means that it is susceptible to shell injection (aka command injection). You can read more about it in 10 common security gotchas in Python and how to avoid them by Anthony Shaw).

Shell injection is an issue any time that

os.system

is receiving unformatted input, like for example when a user can introduce a filename, as in the example above. You can try to execute the script above and give the following input:

dummy; touch harmful_file

This will result in the shell doing:

head -n 1 dummy; touch harmful_file

As a result, the program will first execute

head -n 1 dummy

, as expected, but then it will execute the command

touch harmful_file

to create a file named 'harmful_file'.

Granted that this empty file is not much of a threat, but you can imagine a user adding extra commands to create a file with actual nefarious purposes. This shell injection can also be used to simply transfer or delete information. For example, one could use

;rm -rf ~

;rm -rf /

or any other potentially dangerous command (please do not try these!).

Option 2: subprocess.call (recommended for Python<3.5)

2.1: Not-secure way to run external shell commands

A preferable alternative is

subprocess.call

. As os.sys, the function subprocess.call returns the return value as its output. A naive first approach to subprocess is using the

shell=True

option. This way, the desired command will also be run in a subshell. Note that this is not recommended, as it has the same potential security risks as

os.system

. For example, the code below would be equivalent to the previous

os.system

example.

#!/usr/bin/env python3
import subprocess

# Define command and options wanted
command = "head -n 1 "
# Ask user for file name(s) - SECURITY RISK: susceptible to shell injection
filename = input("Please introduce name of file of interest:\n")
# Run subprocess.call and save return_value
return_value = subprocess.call(command+filename, shell=True)
print('###############')
print('Return value:', return_value)

2.2: Preferred way to run external commands

A preferable way to run

subprocess.call

is by using

shell=False

(it is the default option, so there is no need to specify it). Then, we can simply call

subprocess.call(args)

, where

agrs[0]

contains the command, and

args[1:]

contains all the extra options to the command.

#!/usr/bin/env python3
import subprocess

# Define command and options wanted
command = "head"
options = "-n 1"
# Ask user for file name(s) - now it's safe from shell injection
filename = input("Please introduce name(s) of file(s) of interest:\n")
# Create list with arguments for subprocess.call
args=[]
args.append(command)
args.append(options)
for i in filename.split():
    args.append(i)
# Run subprocess.call and save return_value
return_value = subprocess.call(args)
print('###############')
print('Return value:', return_value)

2.3: Getting standard output:

subprocess.check_output

As mentioned before,

subprocess.call

returns the return value. However, sometimes we might be interested in the standard output returned by the shell command. In this case, we can make use of

subprocess.check_output

To make our script more robust, we can add a try/exclude statement to deal with situations when

check_output

raises an error (in this case, for example, when no file is found).

As shown below, we can use

subprocess.CalledProcessError

in the except clause to deal with the error in a controlled manner and get information about it.

#!/usr/bin/env python3
import subprocess
# Define command and options wanted
command = "head"
options = "-n 1"
# Ask user for file name(s) - now it's safe from shell injection
filename = input("Please introduce name(s) of file(s) of interest:\n")
# Create list with arguments for subprocess.check_output
args=[]
args.append(command)
args.append(options)
for i in filename.split():
    args.append(i)
# Run subprocess.check_output and save command output
try:
    output = subprocess.check_output(args)
    # use decode function to convert to string
    print('###############')
    print('Output:', output.decode("utf-8"))                                                      
# If check_output returns an error: 
except subprocess.CalledProcessError as error:
    print('Error code:', error.returncode, '. Output:', error.output.decode("utf-8"))

Option 3: subprocess.run (recommended since Python 3.5)

The recommended way to execute external shell commands, since Python 3.5, is with the

subprocess.run

function. It is more secure and user-friendly than the previous options discussed.

By default, this function returns an object with the input command and the return code. One can very easily also get the standard output by using the option

capture_output=True

, and finally retrieve the return code and command output using:

output.returncode

and

output.stdout

, respectively.

#!/usr/bin/env python3                                                                            
import subprocess
# Define command and options wanted
command = "head"
options = "-n 1"
# Ask user for file name(s) - now it's safe from shell injection
filename = input("Please introduce name(s) of file(s) of interest:\n")
# Create list with arguments for subprocess.run
args=[]
args.append(command)
args.append(options)
for i in filename.split():
    args.append(i)
# Run subprocess.run and save output object
output = subprocess.run(args,capture_output=True)
print('###############')
print('Return code:', output.returncode)
# use decode function to convert to string
print('Output:',output.stdout.decode("utf-8"))

Final Note on Shell Commands in Python

If you are thinking about using any of the methods discussed here to call an external command, it might be worth considering if there is a standard Python function that allows you to do the same task and will avoid creating a new process. Note that using external commands also makes your program less cross-platform friendly.

External commands might be an attractive option for small tasks if you are familiar with them and the pythonic alternative would imply more learning/implementation time.

However, sticking to Python functions will save you computation time and trouble in the long run, so external commands should be saved for tasks that cannot be achieved with standard Python libraries.

I hope this article helped you to call external commands from your python code!

Featured image by JPuckett Design on Instagram, used with permission.