Python 3.4 多处理递归 Pool.map()

2022-01-12 00:00:00 python 递归 multiprocessing

问题描述

我正在 Ubuntu 14.04 上使用 Python 3.4 进行开发.我试图做递归 Pool.map().在我调用 g() 之后，它挂在那里并且永远不会返回.

I'm developing with Python 3.4 on Ubuntu 14.04. I was trying to do recursive Pool.map(). After I invoke g(), it hangs there and never returns.

import multiprocessing as mp

pool = mp.Pool()

def d(x):
    return x / 2.0


def f(x):
    w = pool.map(d, x)
    return w

def g():
    v = pool.map(f, [[1, 2], [3, 4]])

    print(v)

解决方案

这是不可能的.Pool 对象本身不能安全地在进程之间共享，因此不能在 f 和 g 中使用同一个池.即使您可以这样做，也会很快导致挂起，因为您的池仅限于 cpu_count() 个并发工作人员.一旦你开始递归地创建更多的工人，你最终会得到超过 cpu_count() 个工人，这将永远无法完成；正在运行的工作人员将等待池中排队的任务，但排队的任务将永远无法启动，因为正在运行的任务正在等待.所以你最终陷入僵局.简而言之:不要尝试这样做.

This isn't possible. The Pool object itself can't safely be shared between processes, so the same pool can't be used in both f and g. Even if you could do this, you'd quickly cause a hang, because your pool is limited to cpu_count() concurrent workers. Once you start creating more workers recursively, you'll end up with more than cpu_count() workers, which will never be able to finish; the running workers would be waiting on tasks that are queued up in the pool, but the queued up tasks won't ever be able to start because the running tasks are waiting. So you end up deadlocked. In short: don't try to do this.

相关文章