subprocess.wait() 不等待 Popen 进程完成(使用线程时)?

2022-01-18 00:00:00 python subprocess popen multithreading

问题描述

我在使用 subprocess.Popen() 从我的 python 脚本中使用线程生成同一应用程序的多个实例以使它们同时运行时遇到一些问题.在每个线程中，我使用 popen() 调用运行应用程序，然后通过调用wait() 等待它完成.问题似乎是 wait() 调用实际上并没有等待进程完成.我尝试只使用一个线程，并在进程开始和结束时打印出文本消息.所以线程函数看起来像这样:

I am experiencing some problems when using subprocess.Popen() to spawn several instances of the same application from my python script using threads to have them running simultaneously. In each thread I run the application using the popen() call, and then I wait for it to finish by callingwait(). The problem seems to be that the wait()-call does not actually wait for the process to finish. I experimented by using only one thread, and by printing out text messages when the process starts, and when it finishes. So the thread function would look something like this:

def worker():
    while True:
        job = q.get() # q is a global Queue of jobs
        print('Starting process %d' % job['id'])
        proc = subprocess.Popen(job['cmd'], shell=True)
        proc.wait()
        print('Finished process %d' % job['id'])
        job.task_done()

但即使我只使用一个线程，它也会在出现任何已完成的过程..."消息之前打印出几条正在启动过程..."消息.是否存在 wait() 实际上不等待的情况?我有几个不同的外部应用程序(C++ 控制台应用程序)，这些应用程序又会同时运行多个实例，对于其中一些，我的代码可以工作，但对于其他一些则不行.外部应用程序是否存在某些问题会以某种方式影响对 wait() 的调用?创建线程的代码如下所示:

But even when I only use one thread, it will print out several "Starting process..." messages, before any "Finished process..." message appears. Are there any cases when wait() does not actually wait? I have several different external applications (C++ console applications), which in turn will have several instances running simultaneously, and for some of them my code works, but for others it won't. Could there be some issue with the external applications that somehow affects the call to wait()? The code for creating the threads looks something like this:

for i in range(1):
    t = Thread(target=worker)
    t.daemon = True
    t.start()
q.join() # Wait for the queue to empty

更新 1:我还应该补充一点，对于某些外部应用程序，我有时会得到 -1073471801 的返回码 (proc.returncode).例如，其中一个外部应用程序将在前两次调用 Popen 时给出该返回码，但不是最后两次(当我有四个作业时).

Update 1: I should also add that for some of the external applications I sometimes get a return code (proc.returncode) of -1073471801. For example, one of the external applications will give that return code the first two times Popen is called, but not the last two (when I have four jobs).

更新 2:澄清一下，现在我在队列中有四个作业，它们是四个不同的测试用例.当我运行我的代码时，对于其中一个外部应用程序，前两个 Popen 调用会生成返回码 -1073471801.但是，如果我打印 Popen 调用的确切命令，并在命令窗口中运行它，它会毫无问题地执行.

Update 2: To clear things up, right now I have four jobs in the queue, which are four different test cases. When I run my code, for one of the external applications the first two Popen-calls generate the return code -1073471801. But if I print the exact command which Popen calls, and run it in a command window, it executes without any problems.

解决了！我设法解决了我遇到的问题.我认为问题在于我缺乏线程编程方面的经验.我错过了这样一个事实，即当我创建我的第一个工作线程时，它们会继续存在直到 python 脚本退出.每次我将新项目放入队列时，我都会错误地创建更多工作线程(我为每个要运行的外部程序分批执行此操作).所以当我到达第四个外部应用程序时，我有四个线程同时运行，尽管我只以为我有一个.

Solved! I managed to solve the issues I was having. I think the problem was my lack of experience in threaded programming. I missed the fact that when I had created my first worker threads, they would keep on living until the python script exits. By mistake I created more worker threads each time I put new items on the queue (I do that in batches for every external program I want to run). So by the time I got to the fourth external application, I had four threads running simultaneously even though I only thought I had one.

解决方案

你也可以使用 check_call() 而不是 Popen.check_call() 等待命令完成，即使 shell=True 然后返回作业的退出代码.

You could also use check_call() instead of Popen. check_call() waits for the command to finish, even when shell=True and then returns the exit code of the job.

相关文章