How to process stdout from subprocess but still allow termination at some timeout

Question

There are a number of questions and answers related to Python subprocess and reading of stdout during execution, but still I have not found a solution that fulfills these requirements:

Start a subprocess
Print stdout to terminal, while doing other processing e.g. writing to file
Print stderr to terminal, while doing other processing e.g. writing to file
stdout and stderr must be handled separately
Termination of process if output indicates error or due to some timeout
Linux must be supported, and if possible also Windows (XP/7)

One obstacle is for example that process stdout.read() and stdout.readline() are blocking if no stdout data is generated by the running process, thereby preventing process kill() for termination due to a timeout.

Code for a start:

# Python 3.3 syntax

import sys
import subprocess

def subproc(args, stdout):
    proc = subprocess.Popen(args, stdout=subprocess.PIPE, shell=False, universal_newlines=True)
    while True:  # while is broken by return 
        try:
            outs = ''  # Clear before communication attempt
            outs = proc.communicate(timeout=1)[0]  # Timeout every second to process stdout
        except subprocess.TimeoutExpired:
            pass
        # Processing termination using proc.kill is added later
        sys.__stdout__.write('Shows that proc communicate timeout works\n')
        if outs:
            stdout.write(outs)  # Allow stdout processing
        if proc.returncode is not None:  # Completed execution
            return proc.returncode  # End while and return

class StdHandle:
    def write(self, str):
        # Additional processing ...
        sys.__stdout__.write(str)
    def flush(self):
        pass

rc = subproc(('proc.sh'), stdout=StdHandle())
print('Return code:', rc)

# EOF

http://stackoverflow.com/questions/1606795/catching-stdout-in-realtime-from-subprocess — doctorlove, Aug 19 '13 at 08:58
It should go to normal stdout, but also allow for processing of output, for example to determine termination. — Morten Zilmer, Aug 19 '13 at 09:18
To avoid the blocking calls to `read()`/`readline()` simply use the `select` module. — Bakuriu, Aug 19 '13 at 09:20
@doctorlove: http://stackoverflow.com/questions/1606795 is related, but uses readline(), which then hangs if there is not output from the process, as for what I understand. — Morten Zilmer, Aug 19 '13 at 09:22
@Bakuriu: What function should I look at in the `select` module ? Do you know if it will work both for Linux and Windows ? — Morten Zilmer, Aug 19 '13 at 10:36
@MortenZdk AFAIK `select` is not available on Windows. The `select` function can be used to test whether you can read/write to a given `socket`/`file`. — Bakuriu, Aug 19 '13 at 10:54
Errata: on windows you cannot use `select` with files(which is what you want), but you can use it with `socket`s. — Bakuriu, Aug 19 '13 at 11:00

How to process stdout from subprocess but still allow termination at some timeout

0 Answers0