等待具有多个并行作业的子进程结束

2022-01-18 00:00:00 python subprocess

问题描述

我正在从 python 并行运行一些子进程.我想等到每个子流程都完成.我正在做一个非优雅的解决方案:

I'm running some subprocesses from python in parallel. I want to wait until every subprocess have finished. I'm doing a non elegant solution:

runcodes = ["script1.C", "script2.C"]
ps = []
for script in runcodes:
  args = ["root", "-l", "-q", script]
  p = subprocess.Popen(args)
  ps.append(p)
while True:
  ps_status = [p.poll() for p in ps]
  if all([x is not None for x in ps_status]):
    break

有没有可以处理多个子进程的类?问题是 wait 方法阻塞了我的程序.

is there a class that can handle multiple subprocess? The problem is that the wait method block my program.

更新:我想显示计算过程中的进度:类似于4/7 subprocess finished..."

update: I want to show the progress during the computation: something like "4/7 subprocess finished..."

如果你好奇 root 编译 c++ 脚本并执行它.

If you are curious root compile the c++ script and execute it.

解决方案

如果您的平台不是 Windows，您可能会选择子进程的 stdout 管道.然后，您的应用将被阻止，直到:

If your platform is not Windows, you could probably select against the stdout pipes of your subprocesses. Your app will then block until either:

  • 其中一个已注册的文件描述符有一个 I/O 事件(在这种情况下，我们感兴趣的是子进程的 stdout 管道上的挂断)
  • 投票超时

在 Linux 2.6.xx 中使用 epoll 的非充实示例:

Non-fleshed-out example using epoll with Linux 2.6.xx:

import subprocess
import select

poller = select.epoll()
subprocs = {} #map stdout pipe's file descriptor to the Popen object

#spawn some processes
for i in xrange(5):
    subproc = subprocess.Popen(["mylongrunningproc"], stdout=subprocess.PIPE)
    subprocs[subproc.stdout.fileno()] = subproc
    poller.register(subproc.stdout, select.EPOLLHUP)

#loop that polls until completion
while True:
    for fd, flags in poller.poll(timeout=1): #never more than a second without a UI update
        done_proc = subprocs[fd]
        poller.unregister(fd)
        print "this proc is done! blah blah blah"
        ...  #do whatever
    #print a reassuring spinning progress widget
    ...
    #don't forget to break when all are done

相关文章