Python 进程池非守护进程?

2022-01-12 00:00:00 python multiprocessing pool

问题描述

是否可以创建一个非守护进程的 python 池?我希望一个池能够调用一个内部有另一个池的函数.

Would it be possible to create a python Pool that is non-daemonic? I want a pool to be able to call a function that has another pool inside.

我想要这个，因为守护进程无法创建进程.具体会导致报错:

I want this because deamon processes cannot create process. Specifically, it will cause the error:

AssertionError: daemonic processes are not allowed to have children

例如，考虑 function_a 有一个运行 function_b 的池的场景，该池有一个运行 function_c 的池.此函数链将失败，因为 function_b 正在守护进程中运行，而守护进程无法创建进程.

For example, consider the scenario where function_a has a pool which runs function_b which has a pool which runs function_c. This function chain will fail, because function_b is being run in a daemon process, and daemon processes cannot create processes.

解决方案

multiprocessing.pool.Pool 类在其 __init__ 方法中创建工作进程，使它们成为守护进程并启动它们，并且在它们启动之前无法将它们的 daemon 属性重新设置为 False (之后不再允许).但是您可以创建自己的 multiprocesing.pool.Pool 子类(multiprocessing.Pool 只是一个包装函数)并替换您自己的 multiprocessing.Process 子类，它始终是非守护进程，用于工作进程.

The multiprocessing.pool.Pool class creates the worker processes in its __init__ method, makes them daemonic and starts them, and it is not possible to re-set their daemon attribute to False before they are started (and afterwards it's not allowed anymore). But you can create your own sub-class of multiprocesing.pool.Pool (multiprocessing.Pool is just a wrapper function) and substitute your own multiprocessing.Process sub-class, which is always non-daemonic, to be used for the worker processes.

以下是如何执行此操作的完整示例.重要的部分是顶部的两个类 NoDaemonProcess 和 MyPool 并调用 pool.close() 和 pool.join() 在你的 MyPool 实例最后.

Here's a full example of how to do this. The important parts are the two classes NoDaemonProcess and MyPool at the top and to call pool.close() and pool.join() on your MyPool instance at the end.

#!/usr/bin/env python
# -*- coding: UTF-8 -*-

import multiprocessing
# We must import this explicitly, it is not imported by the top-level
# multiprocessing module.
import multiprocessing.pool
import time

from random import randint


class NoDaemonProcess(multiprocessing.Process):
    # make 'daemon' attribute always return False
    def _get_daemon(self):
        return False
    def _set_daemon(self, value):
        pass
    daemon = property(_get_daemon, _set_daemon)

# We sub-class multiprocessing.pool.Pool instead of multiprocessing.Pool
# because the latter is only a wrapper function, not a proper class.
class MyPool(multiprocessing.pool.Pool):
    Process = NoDaemonProcess

def sleepawhile(t):
    print("Sleeping %i seconds..." % t)
    time.sleep(t)
    return t

def work(num_procs):
    print("Creating %i (daemon) workers and jobs in child." % num_procs)
    pool = multiprocessing.Pool(num_procs)

    result = pool.map(sleepawhile,
        [randint(1, 5) for x in range(num_procs)])

    # The following is not really needed, since the (daemon) workers of the
    # child's pool are killed when the child is terminated, but it's good
    # practice to cleanup after ourselves anyway.
    pool.close()
    pool.join()
    return result

def test():
    print("Creating 5 (non-daemon) workers and jobs in main process.")
    pool = MyPool(5)

    result = pool.map(work, [randint(1, 5) for x in range(5)])

    pool.close()
    pool.join()
    print(result)

if __name__ == '__main__':
    test()

相关文章