How to use `subprocess` command with pipes

Question

I want to use subprocess.check_output() with ps -A | grep 'process_name'. I tried various solutions but so far nothing worked. Can someone guide me how to do it?

related: [How do I use subprocess.Popen to connect multiple processes by pipes?](http://stackoverflow.com/q/295459/4279) — jfs, Jun 16 '13 at 01:34
there is [`psutil`](https://code.google.com/p/psutil/) that allows to get process info in a portable manner. — jfs, Jun 16 '13 at 01:35

score 561 · Accepted Answer · answered Nov 11 '12 at 14:58

561

To use a pipe with the subprocess module, you have to pass shell=True.

However, this isn't really advisable for various reasons, not least of which is security. Instead, create the ps and grep processes separately, and pipe the output from one into the other, like so:

ps = subprocess.Popen(('ps', '-A'), stdout=subprocess.PIPE)
output = subprocess.check_output(('grep', 'process_name'), stdin=ps.stdout)
ps.wait()

In your particular case, however, the simple solution is to call subprocess.check_output(('ps', '-A')) and then str.find on the output.

answered Nov 11 '12 at 14:58

Taymon

23,632
9
61
83

120

+1 for separating the output/input to avoid using `shell=True` – Nicolas Nov 11 '12 at 16:44
7

Don't forget, error ```subprocess.CalledProcessError: Command '('grep', 'process_name')' returned non-zero exit status 1``` just means that nothing was found by grep, so it's normal behaviour. – Serge Jan 27 '15 at 12:17
6

Why do we need the `ps.wait()` for when we already have the output. `ps.wait.__doc__` waits for the child to terminate but the content of the child seems already placed into the `output` variable – Papouche Guinslyzinho Aug 25 '15 at 01:07
It would be nice to use a Python function instead of grep, but I just found that str.find is (now) considered deprecated: https://docs.python.org/2/library/string.html#deprecated-string-functions – MakisH Oct 16 '15 at 19:54
3

@MakisH You're looking at `string.find`, which has been deprecated in favor of `str.find` (i.e., the method `find` on `str` objects). – Taymon Oct 16 '15 at 21:20
7

note: if `grep` dies prematurely; `ps` may hang indefinitely if it produces enough output to fill its OS pipe buffer (because you haven't called `ps.stdout.close()` in the parent). [Swap the starting order, to avoid it](http://stackoverflow.com/a/9164238/4279) – jfs Mar 22 '16 at 17:23
1

What is the reason for using `ps.wait()` instead of `output.wait()`? – aturegano Apr 18 '17 at 10:31
1

@aturegano: Because `output` is a bytestring and does not have a `wait` method. – Taymon Apr 18 '17 at 19:17
1

Some actual explanation for why not to use `shell=True`: https://stackoverflow.com/questions/3172470/actual-meaning-of-shell-true-in-subprocess – User Feb 22 '18 at 07:29
If there is multiple pipe operations should I do `ps.wait()` for each command in order? @Taymon – alper Aug 31 '18 at 13:56
In my case, I was using `yes | accept_licenses` and since `yes` doesn't terminate on it's own, I had to do `time.wait(10); ps.terminate()` instead of `ps.wait()` – Stan Kurdziel Sep 27 '18 at 18:33
Is there a reason you're using tuples instead of lists in the Popen? Why didn't u use `subprocess.Popen(['ps', '-A'], stdout=subprocess.PIPE)` instead of `subprocess.Popen(('ps', '-A'), stdout=subprocess.PIPE)` ? This is the first time I've encountered such usage and I wonder if there's a reason. – eaydin Nov 14 '18 at 18:55
@eaydin: Both `list`s and `tuple`s are sequences, so either will work. The latter aren't mutable and are probably created faster (although I've never tested that assumption). In this particular case the difference likely isn't significant. – martineau Jan 07 '19 at 21:48
as @jfs said, you missed to close the `ps.stdout`, check the man https://docs.python.org/3.5/library/subprocess.html#replacing-shell-pipeline – andras.tim Sep 29 '20 at 13:33

score 73 · Answer 2 · edited Oct 08 '19 at 11:39

73

Or you can always use the communicate method on the subprocess objects.

cmd = "ps -A|grep 'process_name'"
ps = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
output = ps.communicate()[0]
print(output)

The communicate method returns a tuple of the standard output and the standard error.

edited Oct 08 '19 at 11:39

Guillaume Jacquenot

10,118
5
41
48

answered Nov 11 '12 at 16:35

jkalivas

967
1
6
9

7

I think using `communicate` is better than `wait`. [There is](https://docs.python.org/2/library/subprocess.html#subprocess.Popen.wait) such warning: "This will deadlock when using stdout=PIPE and/or stderr=PIPE and the child process generates enough output to a pipe such that it blocks waiting for the OS pipe buffer to accept more data. Use communicate() to avoid that." – Paolo Mar 12 '16 at 16:54
6

To clarify Paolo's comment above, the warning is for wait, not for communicate - i.e. it's the reason he says communicate is better. – EnemyBagJones Dec 02 '16 at 21:16
The output of ps.communicate()[0] in python3 returns a bytes object. – Miguel Ortiz Aug 18 '20 at 14:48
You are reinventing `subprocess.check_output`, not too poorly but unattractively. As the documentation suggests, you should avoid the low-level `Popen` when the library already provides higher-level functions which take care of all this plumbing in a single line of code, often with better behavior for boundary conditions. – tripleee Jul 11 '21 at 08:11
And why are you redirecting standard error to `STDOUT`? – tripleee Jul 11 '21 at 08:12

AlcubierreDrive · Answer 3 · 2013-06-16T01:12:21.613

28

See the documentation on setting up a pipeline using subprocess: http://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline

I haven't tested the following code example but it should be roughly what you want:

query = "process_name"
ps_process = Popen(["ps", "-A"], stdout=PIPE)
grep_process = Popen(["grep", query], stdin=ps_process.stdout, stdout=PIPE)
ps_process.stdout.close()  # Allow ps_process to receive a SIGPIPE if grep_process exits.
output = grep_process.communicate()[0]

edited Jun 16 '13 at 01:12

answered Jun 16 '13 at 00:48

AlcubierreDrive

3,554
2
25
45

2

Upon checking this failed, see the answer below by Taymon for something that works without mucking around – Alvin Dec 02 '13 at 20:08
2

subprocess.check_output doesn't appear to exist in Python 2.6.9 – RightmireM May 29 '14 at 09:53

score 25 · Answer 4 · edited Dec 22 '20 at 06:04

25

Using subprocess.run

import subprocess
    
ps = subprocess.run(['ps', '-A'], check=True, capture_output=True)
processNames = subprocess.run(['grep', 'process_name'],
                              input=ps.stdout, capture_output=True)
print(processNames.stdout)

edited Dec 22 '20 at 06:04

payala

1,297
13
28

answered Nov 12 '20 at 00:50

anaken78

498
4
8

4

NOTE: `capture_output` will only work for Python 3.7.9 and above. – MightyInSpirit Jan 25 '21 at 16:31
1

what does `check` do and what's the purpose of `capture_output`? – CervEd May 05 '21 at 21:25
5

@CervEd Both of these are clearly documented. `capture_output` is a shorthand for the option combination `stdout=supprocess.PIPE, stderr=subprocess.PIPE` and `check=True` raises an error if the subprocess did not return a success (zero) status. – tripleee Jul 11 '21 at 08:17
2

@tripleee they are documented, somewhere in the unwieldy Python documentation, but there's no detail in the answer as to why they are included. `check=True` is for example not strictly necessary but `capture_output=True` is for the answer to work. The reason for using these options should be included as a part of the answer – CervEd Jul 11 '21 at 09:38
I don't necessarily disagree as such, but the answer has a link directly to the pertinent documentation. Perhaps you could have clicked through? – tripleee Jul 11 '21 at 09:41
1

the python documentation is great man, https://docs.python.org/3/library/subprocess.html ctrl-f "capture_output" – Jules G.M. Jul 29 '21 at 20:35
1

A downside to this approach is that `capture_output` will read all of the process's stdout into memory. For small programs like ps, this may be fine, but for larger analysis pipelines this should be avoided. – Eldritch Cheese Aug 25 '21 at 11:19

score 4 · Answer 5 · edited Mar 23 '22 at 19:50

4

You can try the pipe functionality in sh.py:

import sh
print sh.grep(sh.ps("-ax"), "process_name")

edited Mar 23 '22 at 19:50

cyraxjoe

5,428
3
27
42

answered Nov 12 '12 at 05:54

amoffat

658
4
12

3

The link is dead. – tripleee Jul 11 '21 at 08:18
Not anymore, link updated. – cyraxjoe Mar 23 '22 at 19:51

score 4 · Answer 6 · answered Nov 13 '12 at 10:34

4

Also, try to use 'pgrep' command instead of 'ps -A | grep 'process_name'

answered Nov 13 '12 at 10:34

Shooe

77
2

2

if you want get process id, obviously – Shooe Nov 13 '12 at 10:36

score 0 · Answer 7 · edited Oct 08 '19 at 11:50

0

JKALAVIS solution is good, however I would add an improvement to use shlex instead of SHELL=TRUE. below im grepping out Query times

#!/bin/python
import subprocess
import shlex

cmd = "dig @8.8.4.4 +notcp www.google.com|grep 'Query'"
ps = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE,stderr=subprocess.STDOUT)
output = ps.communicate()[0]
print(output)

edited Oct 08 '19 at 11:50

Guillaume Jacquenot

10,118
5
41
48

answered Apr 28 '18 at 01:09

Daniel Smith

49
1
2

1

Why shellx over shell? – AFP_555 Nov 08 '18 at 18:25
11

where is shlex used here ? – 3lokh Nov 25 '19 at 04:01
11

This code _is_ using `shell=True`; shlex is imported, but ignored. There's no improvement here. – Charles Duffy Sep 03 '20 at 16:40

score 0 · Answer 8 · answered Aug 14 '20 at 15:14

0

command = "ps -A | grep 'process_name'"
output = subprocess.check_output(["bash", "-c", command])

answered Aug 14 '20 at 15:14

Brent

3,763
4
24
60

1

Why not use `shell=True` and let that prepend `['sh', '-c']`? Nothing in this code requires bash. (That said, it's significantly better practice to avoid using a shell at all; this use case is a reasonable one, but as soon as arguments start to get parameterized -- like taking the process_name as a parameter -- security concerns come in). – Charles Duffy Sep 03 '20 at 16:39
It's useful in that you don't have to split the string, which gets complicated when you have quoted white space. – Brent Sep 03 '20 at 22:09
Huh? `subprocess.check_output(command, shell=True)` doesn't require you to split the string. `Popen` converts any string into a list containing only that string -- thus, `[command]` -- so with `shell=True` you get `['sh', '-c']` prepended to that list, so you end up with `['sh', '-c', command]`, exactly what your code does here except for the `sh`/`bash` difference. – Charles Duffy Sep 03 '20 at 22:10
...for that matter, if you _did_ try to split the string into a list _as well as_ using `shell=True`, only the first element of that list would be treated as code; you'd get something like `['sh', '-c', 'ps', '-A', '|', 'grep', 'process_name']`. That's not a useful thing to do: when invoked that way, the shell runs `ps` with `$0` being `-A`, `$1` being `|`, etc... but since the command `ps` doesn't look at `$0`, `$1`, etc., all that extra content is simply ignored. – Charles Duffy Sep 03 '20 at 22:13
I wish I could provide a counterexample. This answer exists because I found one, but I don't remember it now. Doing it like this worked perfectly. – Brent Sep 04 '20 at 00:11
1

If you read [`Lib/subprocess.py`](https://github.com/python/cpython/blob/cdbff3527c5b10e79761ad25316b46cd079304bc/Lib/subprocess.py#L1680-L1686), you'll see that there _literally is no difference_ between `subprocess.check_output(["sh", "-c", command])` and `subprocess.check_output(command, shell=True)`. The code is clear and simple -- this is not a place where there can be a devil hiding in the details somewhere. – Charles Duffy Sep 04 '20 at 01:30
Charles is hitting hard with knowledge. – Baskaya Nov 13 '20 at 05:51

score -1 · Answer 9 · answered Dec 02 '19 at 15:06

-1

After Python 3.5 you can also use:

    import subprocess

    f = open('test.txt', 'w')
    process = subprocess.run(['ls', '-la'], stdout=subprocess.PIPE, universal_newlines=True)
    f.write(process.stdout)
    f.close()

The execution of the command is blocking and the output will be in process.stdout.

answered Dec 02 '19 at 15:06

zingi

761
8
19

2

The decoration to write the output to a file is tangential and only complicates this. – tripleee Jul 11 '21 at 08:25

How to use `subprocess` command with pipes

9 Answers9

Linked

Related