When it is only needed to run some command without checking it's output
Also there is check_call method which additionally checks the exit status
getoutput
When it's only needed to get the command's output
Also there is check_output method which additionally checks the exit status
Popen
powerfull method which can run any kind of external command and even to pass it's output to another's command input
Parameters for all subprocess methods are very similar. In fact they are all based on argument list for Popen method. The difference is that some of the methods don't utilize all of them.
These are most important Popen arguments (see it's help for a full list and their meaning):
args: A string, or a sequence of program arguments.
stdin, stdout and stderr: These specify the executed programs' standard input, standard output and standard error file handles, respectively.
shell: If true, the command will be executed through the shell.
cwd: Sets the current directory before the child is executed.
The command which we need to run (args argument, as shown above) should be passed as a list for UNIX (for Windows it doesn't matter):
["ls", "-la", "/tmp"] for command ls -la /tmp
This is due to the security reason to avoid the possibility of command injection.
Executing shell commands that incorporate unsanitized input from an untrusted source makes a program vulnerable to shell injection, a serious security flaw which can result in arbitrary command execution. For this reason, the use of shell=True is strongly discouraged in cases where the command string is constructed from external input:
🪄 Code>>> and 📟Output:
>>>from subprocess import call>>> filename =input("What file would you like to display?\n")What file would you like to display?non_existent; rm -rf />>>call("cat "+ filename, shell=True)# Uh-oh. This will end badly...
This is similar to subprocess.call but additionally will check result code
subprocess.check_call(['ls', '-la'])
This will run command with arguments and wait for command to complete. If the exit code was zero (this mean it was completed successfully) then return, otherwise raise CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute.
🪄 Code:
subprocess.check_call(["touch", "111.txt"])
📟 Output:
0
🪄 Code:
subprocess.check_call(["ls", "111.txt"])
📟 Output:
0
🪄 Code:
subprocess.check_call(["rm", "111.txt"])
📟 Output:
0
🪄 Code:
subprocess.check_call(["ls", "111.txt"])
📟 Output:
CalledProcessErrorTraceback (most recent call last)
<ipython-input-18-bf1b71ed0249> in <module>
----> 1 subprocess.check_call(["ls", "111.txt"])
/opt/conda/lib/python3.7/subprocess.py in check_call(*popenargs, **kwargs)
345 if cmd is None:
346 cmd = popenargs[0]
--> 347 raise CalledProcessError(retcode, cmd)
348 return 0
349
CalledProcessError: Command '['ls', '111.txt']' returned non-zero exit status 2.
🪄 Code:
# Create a temp lock filesubprocess.check_call(["touch", ".lock"])# WORK....# Remove a lock:subprocess.check_call(["rm", ".lock"])
📟 Output:
0
subprocess.getoutput
Easiest way to get the result of running some external command (the contents of it's STDOUT) using shell directly.
Note: only argument is the command to run as string. (Other subprocess method described here use **popenargs mentioned above.
Result 1:
Result 2: -rwxrwxrwx 1 jovyan users 0 Dec 11 10:42 111.txt
Result 3 (status code for <rm -rf 111.txt> command): 0
Result 4: ls: cannot access '111.txt': No such file or directory
subprocess.check_output
This is similar to subprocess.getoutput but:
uses *popenargs arguments
additionally will check result code
subprocess.check_output(['ls', '-la'])
Run command with arguments and return its output. If the exit code was non-zero it raises a CalledProcessError. The CalledProcessError object will have the return code in the returncode attribute and output in the output attribute.
stdin, stdout, stderr – file objects (StringIO is usefull here)
shell – run commands through shell not relying on python implementations. By default it is False
There is a difference in args depending on shell. If it True args must be a string - but this may lead to a seriuos risk.
Executing shell commands that incorporate unsanitized input from an untrusted source makes a program vulnerable to shell injection, a serious security flaw which can result in arbitrary command execution. For this reason, the use of shell=True is strongly discouraged in cases where the command string is constructed from external input:
🪄 Code>>> and 📟Output:
>>>from subprocess import call>>> filename =input("What file would you like to display?\n")What file would you like to display?non_existent; rm -rf /#>>>call("cat "+ filename, shell=True)# Uh-oh. This will end badly...
So it's often recommended to split command to run via using shlex module
Popen is ultimate universal method for running external programs. Popen allows to do everything - reading and writing data to pipes, do pipelines of commands etc.
It allows to watch both stdout and stderr and much more. Just look at syntax:
['cat', 'tmp_file']
'SENDING SOMETHING TO STDIN ---> 2021/04/16'
Pipeline:
! df -ah | grep /notebooks
🪄 Code:
p1 = subprocess.Popen(['df', '-ah'], stdout=subprocess.PIPE)p2 = subprocess.Popen(['grep', '/notebooks'], stdin=p1.stdout, stdout=subprocess.PIPE, text=True)if p1.wait()==0:# Wait for p1 to finish with status 0#p1.stdout.close() # Allow p2 to detect a SIGPIPE if p1 exits p2_output = p2.communicate()[0]print(p2_output)
📟 Output:
/dev/vg1000/lv 7.0T 6.6T 459G 94% /notebooks
Need awk? No problem:
🪄 Code:
import shlex
p1 = subprocess.Popen(['df','-ah'], stdout=subprocess.PIPE)
p2 = subprocess.Popen(['grep', '/notebooks'], stdin=p1.stdout, stdout=subprocess.PIPE)
p3 = subprocess.Popen(shlex.split("awk '{print $5;}'"), stdin=p2.stdout, stdout=subprocess.PIPE, text=True)
if p1.wait() == p2.wait() == 0: # Wait for p2 to finish with status 0
print(p3.communicate()[0])
else:
p3.kill()
📟 Output:
94%
Or - via Python:
🪄 Code:
print(p2_output.split()[4])
📟 Output:
94%
🪄 Code:
import re
print(re.search(r'(\d+%)', p2_output).group(1))
📟 Output:
94%
Another example - let's get how much memory Jupyter Notebook uses.
We can also use context manager for subprocess.Popen to clean resources after running processes:
🪄 Code:
with subprocess.Popen(["ls", "-la", "."], stdout=subprocess.PIPE, text=True) as proc:
print(proc.pid)
print("Alive" if proc.poll() is None else "Terminated")
print("OUTPUT:")
print(proc.communicate()[0])
