Python 多处理库错误(AttributeError:__exit__)

2022-01-12 00:00:00 python multiprocessing pickle with-statement contextmanager

问题描述

使用 pool.map(funct, iterable) 时出现此错误:

Am getting this error when using the pool.map(funct, iterable):

AttributeError: __exit__

没有解释，只是堆栈跟踪到模块内的 pool.py 文件.

No Explanation, only stack trace to the pool.py file within the module.

这样使用:

with Pool(processes=2) as pool:
   pool.map(myFunction, mylist)
   pool.map(myfunction2, mylist2)

我怀疑picklability可能存在问题(python需要pickle，或将列表数据转换为字节流)但我不确定这是真的还是如何调试.

I suspect there could be a problem with the picklability (python needs to pickle, or transform list data into byte stream) yet I'm not sure if this is true or if it is how to debug.

产生此错误的新代码格式:

new format of code that produces this error :

def governingFunct(list):
    #some tasks
    def myFunction():
         # function contents
    with closing(Pool(processes=2)) as pool:
         pool.map(myFunction, sublist)
         pool.map(myFunction2, sublist2)

产生错误:

PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed

解决方案

在 Python 2.x 和 3.0、3.1 和 3.2 中，multiprocessing.Pool() 对象不是上下文管理器.您不能在 with 语句中使用它们.只有在 Python 3.3 及更高版本中，您才能使用它们.来自 Python 3 multiprocessing.Pool() 文档:

In Python 2.x and 3.0, 3.1 and 3.2, multiprocessing.Pool() objects are not context managers. You cannot use them in a with statement. Only in Python 3.3 and up can you use them as such. From the Python 3 multiprocessing.Pool() documentation:

3.3 版中的新功能:池对象现在支持上下文管理协议 - 请参阅上下文管理器类型.__enter__() 返回池对象，__exit__() 调用 terminate().

New in version 3.3: Pool objects now support the context management protocol – see Context Manager Types. __enter__() returns the pool object, and __exit__() calls terminate().

对于早期的 Python 版本，您可以使用 contextlib.closing()，但考虑到这将调用 pool.close()，而不是 pool.terminate().在这种情况下手动终止:

For earlier Python versions, you could use contextlib.closing(), but take into account this'll call pool.close(), not pool.terminate(). Terminate manually in that case:

from contextlib import closing

with closing(Pool(processes=2)) as pool:
    pool.map(myFunction, mylist)
    pool.map(myfunction2, mylist2)
    pool.terminate()

或创建自己的 terminating() 上下文管理器:

or create your own terminating() context manager:

from contextlib import contextmanager

@contextmanager
def terminating(thing):
    try:
        yield thing
    finally:
        thing.terminate()

with terminating(Pool(processes=2)) as pool:
    pool.map(myFunction, mylist)
    pool.map(myfunction2, mylist2)

相关文章