如何打印和显示子进程 stdout 和 stderr 输出而不失真?

2022-01-18 00:00:00 python subprocess

问题描述

也许有人在外面可以帮助我解决这个问题.(我在 SO 上看到了许多与此类似的问题，但没有一个同时处理标准输出和标准错误或处理与我非常相似的情况，因此提出了这个新问题.)

Maybe there's someone out in the ether that can help me with this one. (I have seen a number of similar questions to this on SO, but none deal with both standard out and standard error or deal with a situation quite like mine, hence this new question.)

我有一个 python 函数，它打开一个子进程，等待它完成，然后输出返回码，以及标准输出和标准错误管道的内容.在进程运行时，我还想在填充两个管道时显示它们的输出.我的第一次尝试结果如下:

I have a python function that opens a subprocess, waits for it to complete, then outputs the return code, as well as the contents of the standard out and standard error pipes. While the process is running, I'd like to also display the output of both pipes as they are populated. My first attempt has resulted in something like this:

process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

stdout = str()
stderr = str()
returnCode = None
while True:
    # collect return code and pipe info
    stdoutPiece = process.stdout.read()
    stdout = stdout + stdoutPiece
    stderrPiece = process.stderr.read()
    stderr = stderr + stderrPiece
    returnCode = process.poll()

    # check for the end of pipes and return code
    if stdoutPiece == '' and stderrPiece == '' and returnCode != None:
        return returnCode, stdout, stderr

    if stdoutPiece != '': print(stdoutPiece)
    if stderrPiece != '': print(stderrPiece)

这有几个问题.因为 read() 一直读取到 EOF，所以 while 循环的第一行在子进程关闭管道之前不会返回.

There's a couple problems with this though. Because read() reads until an EOF, the first line of the while loop will not return until the subprocess closes the pipe.

我可以将 read() 替换为 read(int) 但打印输出失真，在读取字符的末尾被切断.我可以用 readline() 作为替代品，但是当同时出现许多输出和错误时，打印输出会失真.

I could replace the read() in favor of read(int) but the printed output is distorted, cut off at the end of the read characters. I could readline() as a replacement, but the printed output is distorted with alternating lines of output and errors when there are many of both that occur at the same time.

也许有一个我不知道的 read-until-end-of-buffer() 变体?还是可以实现?

Perhaps there's a read-until-end-of-buffer() variant that I'm not aware of? Or maybe it can be implemented?

也许最好按照 sys.stdout 包装器">回答另一个帖子?然而，我只想在这个函数中使用包装器.

Maybe it's best to implement a sys.stdout wrapper as suggested in this answer to another post? I would only want to use the wrapper in this function, however.

社区还有其他想法吗?

感谢您的帮助！:)

编辑:该解决方案确实应该是跨平台的，但如果您有不跨平台的想法，请将它们分享出去以保持头脑风暴的进行.

EDIT: The solution really should be cross-platform, but if you have ideas that aren't, please share them away to keep the brainstorming going.

对于我的另一个 python 子进程头抓手，看看我的另一个问题 在计时中考虑子进程开销.

For another one of my python subprocess head scratchers, take a look at another of my questions on accounting for subprocess overhead in timing.

解决方案

使用 fcntl.fcntl，并使用 select.select 等待数据在任一管道中可用.例如:

Make the pipes non-blocking by using fcntl.fcntl, and use select.select to wait for data to become available in either pipe. For example:

# Helper function to add the O_NONBLOCK flag to a file descriptor
def make_async(fd):
    fcntl.fcntl(fd, fcntl.F_SETFL, fcntl.fcntl(fd, fcntl.F_GETFL) | os.O_NONBLOCK)

# Helper function to read some data from a file descriptor, ignoring EAGAIN errors
def read_async(fd):
    try:
        return fd.read()
    except IOError, e:
        if e.errno != errno.EAGAIN:
            raise e
        else:
            return ''

process = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
make_async(process.stdout)
make_async(process.stderr)

stdout = str()
stderr = str()
returnCode = None

while True:
    # Wait for data to become available 
    select.select([process.stdout, process.stderr], [], [])

    # Try reading some data from each
    stdoutPiece = read_async(process.stdout)
    stderrPiece = read_async(process.stderr)

    if stdoutPiece:
        print stdoutPiece,
    if stderrPiece:
        print stderrPiece,

    stdout += stdoutPiece
    stderr += stderrPiece
    returnCode = process.poll()

    if returnCode != None:
        return (returnCode, stdout, stderr)

请注意，fcntl 仅适用于类 Unix 平台，包括 Cygwin.

Note that fcntl is only available on Unix-like platforms, including Cygwin.

如果您需要它在没有 Cygwin 的情况下在 Windows 上工作，这是可行的，但要困难得多.您必须:

If you need it to work on Windows without Cygwin, it's doable, but it's much, much tougher. You'll have to:

  • 使用 pywin32 库调用本机 Win32 API
  • 使用 SetNamedPipeHandleState 与 PIPE_NOWAIT 使 stdout 和 stderr 管道非阻塞
  • 使用 WaitForMultipleObjects 而不是 select 等待数据可用
  • 使用 ReadFile 读取数据
  • Use the pywin32 library to call through to the native Win32 API
  • Use SetNamedPipeHandleState with PIPE_NOWAIT to make the stdout and stderr pipes non-blocking
  • Use WaitForMultipleObjects instead of select to wait for data to become available
  • Use ReadFile to read the data

相关文章