python StringIO doesn't work as file with subrpocess.call()

Question

I'm working with subprocess package to call some external console commands from a python script, and I need to pass file handlers to it to get stdout and stderr back separately. The code looks like this roughly:

import subprocess

stdout_file = file(os.path.join(local_path, 'stdout.txt'), 'w+')
stderr_file = file(os.path.join(local_path, 'stderr.txt'), 'w+')

subprocess.call(["somecommand", "someparam"], stdout=stdout_file, stderr=stderr_file)

This works fine and txt files with relevant output are getting created. Yet it would be nicer to handle these outputs in memory omitting files creation. So I used StringIO package to handle it this way:

import subprocess
import StringIO

stdout_file = StringIO.StringIO()
stderr_file = StringIO.StringIO()

subprocess.call(["somecommand", "someparam"], stdout=stdout_file, stderr=stderr_file)

But this doesn't work. Fails with:

  File "./test.py", line 17, in <module>
    subprocess.call(["somecommand", "someparam"], stdout=stdout_file, stderr=stderr_file)
  File "/usr/lib/python2.7/subprocess.py", line 493, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 672, in __init__
    errread, errwrite) = self._get_handles(stdin, stdout, stderr)
  File "/usr/lib/python2.7/subprocess.py", line 1063, in _get_handles
    c2pwrite = stdout.fileno()
AttributeError: StringIO instance has no attribute 'fileno'

I see that it's missing some parts of the native file object and fails because of that.

So the question is more educational than practical - why these parts of file interface are missing from StringIO and are there any reasons why this cannot be implemented?

hmm, thing is that `subprocess.check_output` throws out all output as one string, while I need to separate **stdout** from **stderr** — alexykot, Oct 16 '13 at 16:58
hmm, found a workaround using Popen instead of subprocess. Explained here http://stackoverflow.com/questions/10103551/passing-data-to-subprocess-check-output — alexykot, Oct 16 '13 at 17:01
selffix - Popen is a part of subprocess, so instead of subprocess.call() or subprocess.check_output() — alexykot, Oct 16 '13 at 17:15
See http://stackoverflow.com/questions/5903501/attributeerror-stringio-instance-has-no-attribute-fileno. Yes, this is a bug in the standard library. — Charles Merriam, Mar 05 '17 at 17:34
Original post was done in 2013. Solution for this problem is not on my priority list anymore, but thanks anyway. — alexykot, Mar 07 '17 at 16:42

Vanessa Phipps · Answer 1 · 2017-06-08T03:01:32.657

As you said in your comment, Popen and Popen.communicate are the right solution here.

A bit of background: real file objects have file descriptors, which is the fileno attribute StringIO objects are missing. They're just ordinary integers: you may be familiar with file descriptors 0, 1 and 2, which are stdin, stdout and stderr, respectively. If a process opens more files, they're assigned 3, 4, 5, etc.. You can take a look at a process's current file descriptors with lsof -p.

So, why can't StringIO objects have file descriptors? In order to get one, it'd need to either open a file or open a pipe*. Opening a file wouldn't make sense, since not opening files is the whole point of using StringIO in the first place.

And opening a pipe also wouldn't make sense, even though they live in memory like StringIO objects do. They're for communication, not storage: seek, truncate, and len have no meaning at all for pipes, and read and write behave very differently than they do for files. When you read from a pipe, the returned data is deleted from the pipe's buffer, and if that (relatively small) buffer is full when you try to write, your process will hang until something reads from the pipe to free up buffer space.

So if you want to use a string as stdin, stdout or stderr for a subprocess, StringIO won't cut it but Popen.communicate is perfect. As stated above (and warned about in subprocess's docs), reading from and writing to pipes correctly is complicated. Popen handles that complexity for you.

* I guess I could theoretically imagine a third kind of file descriptor corresponding to a memory region shared between processes? Not really sure why that doesn't exist. But eh, I'm not a kernel developer, so I'm sure there's a reason.

[The documentation](https://docs.python.org/3.6/library/subprocess.html#frequently-used-arguments) is misleading. It says that the value of `stdout` can be "*an existing [file object](https://docs.python.org/3.6/glossary.html#term-file-object)*", not mentioning that it requires the file object to have a `fileno`. — Franklin Yu, Jun 06 '17 at 21:39

score -2 · Answer 2 · answered Oct 16 '13 at 17:02

-2

I think that you are expecting some other process to know how to read the memory as a stream from your main process. Perhaps if you can pipe your stream into the standard input and pipe the standard output into your stream you might be successful.

answered Oct 16 '13 at 17:02

Fred Mitchell

2,061
1
19
28

python StringIO doesn't work as file with subrpocess.call()

2 Answers2