在python中控制用于调用外部命令的子进程数

2022-01-12 00:00:00 python subprocess multiprocessing parallel-processing

问题描述

我了解使用 subprocess 是调用外部命令的首选方式.

I understand using subprocess is the preferred way of calling external command.

但是，如果我想并行运行多个命令，但要限制生成的进程数怎么办?困扰我的是我无法阻止子进程.例如，如果我调用

But what if I want to run several commands in parall, but limit the number of processes being spawned? What bothers me is that I can't block the subprocesses. For example, if I call 

subprocess.Popen(cmd, stderr=outputfile, stdout=outputfile)

然后该过程将继续，无需等待 cmd 完成.因此，我无法将其包装在 multiprocessing 库的工作人员中.

Then the process will continue, without waiting for cmd to finish. Therefore, I can't wrap it up in a worker of multiprocessing library.

例如，如果我这样做:

def worker(cmd): 
    subprocess.Popen(cmd, stderr=outputfile, stdout=outputfile);

pool = Pool( processes = 10 );
results =[pool.apply_async(worker, [cmd]) for cmd in cmd_list];
ans = [res.get() for res in results];

然后每个工作人员将在生成子进程后完成并返回.所以我不能通过使用Pool 来真正限制subprocess 生成的进程数.

then each worker will finish and return after spawning a subprocess. So I can't really limit the number of processes generated by subprocess by using Pool.

限制子进程数量的正确方法是什么?

What's the proper way of limiting the number of subprocesses?

解决方案

如果要等待命令完成，可以使用subprocess.call.有关详细信息，请参阅 pydoc 子进程.

You can use subprocess.call if you want to wait for the command to complete. See pydoc subprocess for more information.

您也可以调用 Popen.wait 你的工人中的方法:

You could also call the Popen.wait method in your worker:

def worker(cmd): 
    p = subprocess.Popen(cmd, stderr=outputfile, stdout=outputfile);
    p.wait()

相关文章