Python 多处理从不加入

2022-01-12 00:00:00 python process multithreading multiprocessing pool

问题描述

我正在使用 multiprocessing，特别是一个 Pool 来分拆几个线程"来完成我拥有的一堆慢速工作.但是，由于某种原因，我无法让主线程重新加入，即使所有的孩子似乎都已经死了.

I'm using multiprocessing, and specifically a Pool to spin off a couple of 'threads' to do a bunch of slow jobs that I have. However, for some reason, I can't get the main thread to rejoin, even though all of the children appear to have died.

已解决:看来这个问题的答案是只启动多个 Process 对象，而不是使用 Pool.目前尚不清楚为什么，但我怀疑剩余的进程是池的管理器，并且当进程完成时它并没有死亡.如果其他人有这个问题，这就是答案.

Resolved: It appears the answer to this question is to just launch multiple Process objects, rather than using a Pool. It's not abundantly clear why, but I suspect the remaining process is a manager for the pool and it's not dying when the processes finish. If anyone else has this problem, this is the answer.

主线程

pool = Pool(processes=12,initializer=thread_init)
for x in xrange(0,13):
    pool.apply_async(thread_dowork)
pool.close()
sys.stderr.write("Waiting for jobs to terminate
")
pool.join()

xrange(0,13) 比进程数多一，因为我认为我有一个关闭，并且一个进程没有得到工作，所以是'不会死，我想强迫它去工作.我也试过 12 次.

The xrange(0,13) is one more than the number of processes because I thought I had an off by one, and one process wasn't getting a job, so wasn't dying and I wanted to force it to take a job. I have tried it with 12 as well.

多处理函数

def thread_init():
    global log_out
    log_out = open('pool_%s.log'%os.getpid(),'w')
    sys.stderr = log_out
    sys.stdout = log_out
    log_out.write("Spawned")
    log_out.flush()
    log_out.write(" Complete
")
    log_out.flush()


def thread_dowork():
    log_out.write("Entered function
")
    log_out.flush()
    #Do Work
    log_out.write("Exiting ")
    log_out.flush()
    log_out.close()
    sys.exit(0)

所有 12 个孩子的日志文件的输出是:

The output of the logfiles for all 12 children is:

Spawned
Complete
Entered function
Exiting

主线程打印等待作业终止"，然后就坐在那里.

The main thread prints 'Waiting for jobs to terminate', and then just sits there.

top 仅显示脚本的一份副本(我相信是主要的).htop 显示两个副本，一个是从顶部开始的副本，另一个是其他副本.根据它的 PID，它也不是孩子.

top shows only one copy of the script (the main one I believe). htop shows two copies, one of which is the one from top, and the other one of which is something else. Based on its PID, it's none of the children either.

有人知道我不知道的事情吗?

Does anyone know something I don't?

解决方案

我真的没有答案，但我阅读了 Apply_async 的文档，这似乎与您所说的问题背道而驰......

I don't really have an answer but I read the docs for Apply_async and it seems counter to your stated problem...

回调应该立即完成，否则线程处理结果将被阻止.

Callbacks should complete immediately since otherwise the thread which handles the results will get blocked.

我不熟悉池，但在我看来，您的用例可以通过 本周 Python 模块

I'm not familiar with the Pool but it seems to me that your use-case could easily be handled by this recipe on Python Module of the Week

相关文章