如何从 Python 迭代器提供子进程的标准输入?

2022-01-18 00:00:00 python subprocess io

问题描述

我正在尝试使用 Python 中的 subprocess 模块与以流方式读取标准输入并写入标准输出的进程进行通信.我想让子进程从产生输入的迭代器中读取行，然后从子进程中读取输出行.输入和输出线之间可能没有一一对应的关系.如何从返回字符串的任意迭代器中提供子进程?

I am trying to use the subprocess module in Python to communicate with a process that reads standard input and writes standard output in a streaming fashion. I want to have the subprocess read lines from an iterator that produces the input, and then read output lines from the subprocess. There may not be a one-to-one correspondence between input and output lines. How can I feed a subprocess from an arbitrary iterator that returns strings?

这里有一些示例代码给出了一个简单的测试用例，以及我尝试过的一些方法由于某种原因不起作用:

Here is some example code that gives a simple test case, and some methods I have tried that don't work for some reason or other:

#!/usr/bin/python
from subprocess import *
# A really big iterator
input_iterator = ("hello %s
" % x for x in xrange(100000000))

# I thought that stdin could be any iterable, but it actually wants a
# filehandle, so this fails with an error.
subproc = Popen("cat", stdin=input_iterator, stdout=PIPE)

# This works, but it first sends *all* the input at once, then returns
# *all* the output as a string, rather than giving me an iterator over
# the output. This uses up all my memory, because the input is several
# hundred million lines.
subproc = Popen("cat", stdin=PIPE, stdout=PIPE)
output, error = subproc.communicate("".join(input_iterator))
output_lines = output.split("
")

那么当我从迭代器的标准输出中逐行读取时，如何让我的子进程逐行读取呢?

So how can I have my subprocess read from an iterator line by line while I read from its stdout line by line?

解决方案

简单的方法似乎是从子进程分叉并提供输入句柄.任何人都可以详细说明这样做的任何可能的缺点吗?或者是否有 python 模块可以让它更容易和更安全?

The easy way seems to be to fork and feed the input handle from the child process. Can anyone elaborate on any possible downsides of doing this? Or are there python modules that make it easier and safer?

#!/usr/bin/python
from subprocess import *
import os

def fork_and_input(input, handle):
    """Send input to handle in a child process."""
    # Make sure input is iterable before forking
    input = iter(input)
    if os.fork():
        # Parent
        handle.close()
    else:
        # Child
        try:
            handle.writelines(input)
            handle.close()
        # An IOError here means some *other* part of the program
        # crashed, so don't complain here.
        except IOError:
            pass
        os._exit()

# A really big iterator
input_iterator = ("hello %s
" % x for x in xrange(100000000))

subproc = Popen("cat", stdin=PIPE, stdout=PIPE)
fork_and_input(input_iterator, subproc.stdin)

for line in subproc.stdout:
    print line,

相关文章