There's is no shortage of questions on SO about getting real-time stdout from a process with python. We also need it to be non-blocking, so:
while True:
line = p.stdout.readline()
if not line: break
...
...is not an option. Edit: Actually testing this solution now, it doesn't even work.
It's not even quite clear why that works and non-blocking versions using bufsize=1 don't. This is not a duplicate of Getting realtime output using subprocess, as none of the solutions suggested there seem to apply to a non-blocking version of the function. I've tested them. Not sure why bufsize=1 does not work.
There doesn't actually seem to be any solution now, because as the author of that answer cited above points out, it might have been a bug to begin with.
There are other questions about non-blocking versions of the the function too, but no question I've found asks for / receives answers to the problem of both non-blocking and forced unbuffering.
Here's a non-blocking func to run a command and poll for output:
import subprocess
from set_interval import set_interval
def call_command(cmd, callback=None):
try:
p = subprocess.Popen(cmd, stderr=subprocess.PIPE, stdout=subprocess.PIPE, encoding='utf-8', bufsize=1)
def polling():
for line in iter(p.stdout.readline, ''):
if p.returncode != 0 or (not line and p.poll() is not None):
p.kill()
return False # stop interval
else:
if callback is not None:
callback(line.strip())
else:
print(line.strip())
return True # continue interval
set_interval(polling, 1)
if p.returncode != 0:
err = p.communicate()
print(err[1].strip())
except FileNotFoundError as e:
print(f'{e} "{cmd[0]}"')
set_interval.py, referenced above:
import threading
def set_interval(func, sec):
result = func()
def func_wrapper():
if result is True:
set_interval(func, sec)
t = threading.Timer(sec, func_wrapper)
t.start()
The challenge we're facing is let's say the program generating output isn't flushing the output buffer:
output_unflushed.py:
from time import sleep
for i in range(10):
print('i:', i)
sleep(1)
Well if you do call_command('python output_unflushed.py'.split(' ')) you'll get no output for 10 seconds, then you'll get it all at once. This is because Python is just buffering all the stdout. As @radekholy24 pointed out we can fix that easily by modifying output_unflushed.py:
from time import sleep
import sys
for i in range(10):
print('i:', i)
sys.stdout.flush()
sleep(1)
Unfortunately, we don't control the source of every process we want to run. Maybe we do, but not always/easily. For example, git clone seems to suffer from this same issue.
There are binaries that can be used to solve this (in theory, I tried the winpty solution and it didn't work for me), but that adds a lot of complexity:
- unbuffer, citing this answer
- winpty, citing this answer.
This feels like major overkill, complicating my code, running it through another program, having to involve mingw dlls, etc. No thanks.
Also PYTHONUNBUFFERED=1 and similar process-specific solutions aren't what I'm interested in.
Is there any way to accomplish this with Python?
This must be possible in universally applicable way, since obviously the command prompts all manage it. Whether it's possible in Python (with non-blocking code) is a different story...
