Edit subprocess.Popen( linux command ) on Windows10

Question

I'm studying image matching with Superpoint(https://github.com/rpautrat/SuperPoint).

I tried to run the codes written in Usage area. Unfortunately, I'm using this code on Windows10 and Superpoint written for linux so I got a lot of error... :(

Especially, https://github.com/rpautrat/SuperPoint/blob/master/superpoint/utils/stdout_capturing.py cause critical error to me.

There is a part of subprocess.Popen for write the log file with file descriptor. I want to get log file result as same as result gotten from the linux os.

The code is below.... and I think that tee command is not working on Windows that's why this part cause the error.

I hope edit this part for the run whole code for studying Superpoint.

Please let me know the solution for this file descriptor on Windows10 problem.

Thanks a lot for your kindness help!

#!/usr/bin/env python
# coding=utf-8
from __future__ import division, print_function, unicode_literals
import os
import sys
import subprocess
from threading import Timer
from contextlib import contextmanager

'''
Based on sacred/stdout_capturing.py in project Sacred
https://github.com/IDSIA/sacred
'''


def flush():
    """Try to flush all stdio buffers, both from python and from C."""
    try:
        sys.stdout.flush()
        sys.stderr.flush()
    except (AttributeError, ValueError, IOError):
        pass  # unsupported


# Duplicate stdout and stderr to a file. Inspired by:
# http://eli.thegreenplace.net/2015/redirecting-all-kinds-of-stdout-in-python/
# http://stackoverflow.com/a/651718/1388435
# http://stackoverflow.com/a/22434262/1388435
@contextmanager
def capture_outputs(filename):
    """Duplicate stdout and stderr to a file on the file descriptor level."""
    # with NamedTemporaryFile(mode='w+') as target:
    with open(filename, 'a+') as target:
        original_stdout_fd = 1
        original_stderr_fd = 2
        target_fd = target.fileno()

        # Save a copy of the original stdout and stderr file descriptors
        saved_stdout_fd = os.dup(original_stdout_fd)
        saved_stderr_fd = os.dup(original_stderr_fd)

        tee_stdout = subprocess.Popen(
            ['tee', '-a', '/dev/stderr'], start_new_session=True,
            stdin=subprocess.PIPE, stderr=target_fd, stdout=1)
        tee_stderr = subprocess.Popen(
            ['tee', '-a', '/dev/stderr'], start_new_session=True,
            stdin=subprocess.PIPE, stderr=target_fd, stdout=2)

        flush()
        os.dup2(tee_stdout.stdin.fileno(), original_stdout_fd)
        os.dup2(tee_stderr.stdin.fileno(), original_stderr_fd)

        try:
            yield
        finally:
            flush()

            # then redirect stdout back to the saved fd
            tee_stdout.stdin.close()
            tee_stderr.stdin.close()

            # restore original fds
            os.dup2(saved_stdout_fd, original_stdout_fd)
            os.dup2(saved_stderr_fd, original_stderr_fd)

            # wait for completion of the tee processes with timeout
            # implemented using a timer because timeout support is py3 only
            def kill_tees():
                tee_stdout.kill()
                tee_stderr.kill()

            tee_timer = Timer(1, kill_tees)
            try:
                tee_timer.start()
                tee_stdout.wait()
                tee_stderr.wait()
            finally:
                tee_timer.cancel()

            os.close(saved_stdout_fd)
            os.close(saved_stderr_fd)

Answer 1

I'd go with this implementation: https://stackoverflow.com/a/17942748/5430833

First of all, 'tee' is a linux utility, unless you've got some port installed. /dev/stderr is also a unix descriptor for error output stream. As per this answer: cmd.exe equivalent of /dev/stdout, to write to stdout 'as a file', there's no such descriptor equivalent for windows.

If you want your solution to be portable, I suggest sticking to python and avoiding Popen unless you are sure that process you are starting works on both platforms.

Edit: okay, forgive me as I haven't read the question carefully. So, you want some kind of a 'deamon' process which will capture all stdout and stderr. If you can live without output in the console, I'd go with a simple solution: comment the 'capture_outputs' content out and at your application boot (your test file or some kind of a main, before any SuperPoint code executes) redirect the streams:

sys.stdout = open('out.dat', 'w')

And somewhere at exit:

sys.stdout.close()

You could wrap your logic with a ContextManager so that in case of some internal exception files are closed properly.

I believe you could do the redirect within the 'capture_outputs' as well, but I advise to keep it simple unless we're sure it works.