How to Run External Programs in Python

Modern cloud services, such as Hostman, have become a part of everyday life. Remote access and control are the foundation of today’s digital world. In this article, we’ll show you how to run external programs using different modules in Python 3.

The built-in subprocess module is used to launch external programs from Python and capture their input or output. This module allows you to run and control the execution of various programs available on your computer.

To launch a program using the subprocess module, there are a few functions you can use, such as subprocess.run (which replaced subprocess.call() in Python 3.5) and subprocess.Popen.

Syntax of subprocess.run

subprocess.run(args, *, stdin, input, stdout, stderr, 
               capture_output, shell, cwd, timeout, 
               check, encoding, errors, text, env, 
               universal_newlines)

• args – Required. Specifies the program to be executed with its arguments.

• stdin, input, stdout, and stderr – These parameters handle the input/output streams of the process. stdout represents the output (result), and stderr handles errors. By default, they are set to None.

• capture_output – Default is False. If set to True, it captures the process's output.

• shell – If the program and its arguments are passed as a single string, you need to set this to True. By default, it’s False.

• cwd – Used to specify the absolute path to the directory where the program is run.

• timeout – Time in seconds after which the process will be terminated, triggering an exception.

• check – If set to True, it raises an exception if there were any errors during execution. Default is False.

• encoding – Handles the decoding of the output.

• errors – If specified, encoding errors will trigger an exception.

• text, universal_newlines – Text modes for input, output, and error streams. Default is False.

• env – Specifies the environment variables for the new process.

The function returns a CompletedProcess object containing the results. With its attributes, you can retrieve the passed arguments, execution result code, output, or any errors that occurred. When using this function, be cautious with code from untrusted sources, especially if running with admin privileges, as it can execute anything.

Before jumping into examples, let’s import the necessary modules:

from subprocess import run, Popen, PIPE
from sys import executable

The executable variable in Python is the absolute path to the Python interpreter, which will be needed to execute code.

Let's look at a simple example using CompletedProcess.

run("echo 'Subprocesses are cool!'", shell=True)

Output:

Subprocesses are cool!

The result will be a CompletedProcess object:

CompletedProcess(args="echo 'Subprocesses are cool!'", returncode=0)

Here, we see the arguments we passed and the status code (returncode). A return code of 0 indicates that the process was completed successfully.

In this example, we simply launched an external process and got confirmation that it was completed successfully. But what if we want to capture not just the status code, but also some output data? For this, we need to set the capture_output = True parameter:

result = run('ping localhost', shell=True, capture_output=True, encoding='cp737')

In this example, we send a ping request to localhost to check the connection. The encoding parameter specifies that we must decode the result; otherwise, we’ll receive it as a byte string.

To get the program's output and any errors, we can execute:

print("stdout:", result.stdout)
print("stderr:", result.stderr)

Output:

stdout:
Pinging DESKTOP-*** [::1] with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

stderr:

If an error occurs during execution, the subprocess will return it in result.stderr. In this case, no errors occurred.

This shows how to capture both the output and errors when executing external processes using Python's subprocess module.

Exception Handling

Let’s run the following code:

run([executable, "-c", "raise ValueError('It seems there is a mistake!')"], capture_output=True, encoding='cp737')

The -c element is a Python command-line option that allows us to pass a complete program.

This will work without issues, but if we try to print stdout, we’ll see that it is empty (unlike stderr).

Output:

stdout:
stderr: Traceback (most recent call last):
  File "<string>", line 1, in <module>
ValueError: It seems there is a mistake!

If we call the check_returncode method, we’ll see the following:

CalledProcessError: Command '['D:\\***\\python.exe', '-c', "raise ValueError('It seems there is a mistake!')"]' returned non-zero exit status 1.

We encountered an error. To raise an exception when a subprocess fails, you need to specify check=True. In this case, the code won't run, and you’ll immediately know that a problem occurred during the process.

Setting a Timeout

We can set a time limit for the subprocess to prevent a misbehaving program from running indefinitely. This is done using the timeout parameter:

run('ping hostman.com', shell=True, timeout=5, capture_output=True, encoding='cp737')

If the command completes successfully, we get the following output:

Pinging hostman.com [188.114.96.3] with 32 bytes of data:
Reply from 188.114.96.3: bytes=32 time=17ms TTL=56
Reply from 188.114.96.3: bytes=32 time=17ms TTL=56
Reply from 188.114.96.3: bytes=32 time=17ms TTL=56
Reply from 188.114.96.3: bytes=32 time=17ms TTL=56

However, if the command doesn’t finish in time, we get the following exception:

TimeoutExpired: Command 'ping hostman.com' timed out after 1 seconds

Running Installed Programs

With subprocesses, we can also run installed programs. For example:

run("notepad.exe")

This will open Notepad.

We can also pass some input using stdin and the input parameter. Let’s execute the following code:

run([executable, "-c", "from sys import stdin; print(stdin.read())"], input="Hello subprocess!", capture_output=True, encoding='cp737')

The result will be:

Hello subprocess!

The input parameter can be used not only to pass specific text but also to pass the output of a program.

subprocess.run Function

The run function from the subprocess module is high-level. You can use the Popen class directly if your task requires more flexibility. It has a similar syntax but provides additional parameters.

Let’s create two subprocesses. In the first, we’ll get the contents of a directory, and in the second, we’ll sort the results. To use the output from the first subprocess, we specify stdout=PIPE, which can be passed as an argument for stdin, stdout, or stderr.

To connect two processes, use the communicate() method:

p1 = Popen('dir', shell=True, stdin=None, stdout=PIPE, stderr=PIPE)
p2 = Popen('sort /R', shell=True, stdin=p1.stdout)
p1.stdout.close()
out, err = p2.communicate()

The result of this code will be the sorted files in the directory:

21.03.2022  12:35    <DIR>       ..
21.03.2022  12:35    <DIR>      .
21.03.2022  12:35           256 test1.py
21.03.2022  12:34           255 test3.py
21.03.2022  12:21           247 test5.py
21.03.2022  12:21           247 test4.py
21.03.2022  12:21           247 test2.py
21.03.2022  12:21           247 test.py

Useful Popen Methods

• poll() – Checks if the subprocess has finished. Returns the return code or None if the process is still running.

• wait(timeout=None) – Waits for the subprocess to finish. If timeout is specified, it will raise a TimeoutExpired exception after the given time.

• terminate() – Stops the process.

Conclusion

In this tutorial, we looked at how to run external programs using Python. This is a standard feature of the Python library that allows you to execute external programs and capture their output.

On our app platform you can find Python applications, such as Celery, Django, FastAPI and Flask.