使用 subprocess.Popen 的非常大的输入和管道

I have pretty simple problem. I have a large file that goes through three steps, a decoding step using an external program, some processing in python, and then recoding using another external program. I have been using subprocess.Popen() to try to do this in python rather than forming unix pipes. However, all the data are buffered to memory. Is there a pythonic way of doing this task, or am I best dropping back to a simple python script that reads from stdin and writes to stdout with unix pipes on either side?

import os, sys, subprocess

def main(infile,reflist):
    print infile,reflist
    samtoolsin = subprocess.Popen(["samtools","view",infile],
                                  stdout=subprocess.PIPE,bufsize=1)
    samtoolsout = subprocess.Popen(["samtools","import",reflist,"-",
                                    infile+".tmp"],stdin=subprocess.PIPE,bufsize=1)
    for line in samtoolsin.stdout.read():
        if(line.startswith("@")):
            samtoolsout.stdin.write(line)
        else:
            linesplit = line.split("	")
            if(linesplit[10]=="*"):
                linesplit[9]="*"
            samtoolsout.stdin.write("	".join(linesplit))

解决方案

Popen 有一个 bufsize 参数，它将限制内存中缓冲区的大小.如果您根本不希望内存中的文件，您可以将文件对象作为 stdin 和 stdout 参数传递.来自 子进程文档: