具有多处理功能的 Python 装饰器失败

2022-01-12 00:00:00 python multiprocessing decorator

问题描述

我想在随后将传递给多处理池的函数上使用装饰器.但是，代码因PicklingError: Can't pickle : attribute lookup __builtin__.function failed"而失败.我不太明白为什么它在这里失败.我确信这很简单，但我找不到.下面是一个最小的工作"示例.我认为使用 functools 函数就足以让它工作.

I would like to use a decorator on a function that I will subsequently pass to a multiprocessing pool. However, the code fails with "PicklingError: Can't pickle : attribute lookup __builtin__.function failed". I don't quite see why it fails here. I feel certain that it's something simple, but I can't find it. Below is a minimal "working" example. I thought that using the functools function would be enough to let this work.

如果我注释掉函数装饰，它可以正常工作.我在这里误解的 multiprocessing 是什么?有什么办法可以做到吗?

If I comment out the function decoration, it works without an issue. What is it about multiprocessing that I'm misunderstanding here? Is there any way to make this work?

编辑:在添加了一个可调用的类装饰器和一个函数装饰器之后，函数装饰器按预期工作.可调用的类装饰器继续失败.防止它被腌制的可调用类版本是什么?

Edit: After adding both a callable class decorator and a function decorator, it turns out that the function decorator works as expected. The callable class decorator continues to fail. What is it about the callable class version that keeps it from being pickled?

import random
import multiprocessing
import functools

class my_decorator_class(object):
    def __init__(self, target):
        self.target = target
        try:
            functools.update_wrapper(self, target)
        except:
            pass

    def __call__(self, elements):
        f = []
        for element in elements:
            f.append(self.target([element])[0])
        return f

def my_decorator_function(target):
    @functools.wraps(target)
    def inner(elements):
        f = []
        for element in elements:
            f.append(target([element])[0])
        return f
    return inner

@my_decorator_function
def my_func(elements):
    f = []
    for element in elements:
        f.append(sum(element))
    return f

if __name__ == '__main__':
    elements = [[random.randint(0, 9) for _ in range(5)] for _ in range(10)]
    pool = multiprocessing.Pool(processes=4)
    results = [pool.apply_async(my_func, ([e],)) for e in elements]
    pool.close()
    f = [r.get()[0] for r in results]
    print(f)

解决方案

问题是 pickle 需要有一些方法来重新组装你 pickle 的所有东西.请参阅此处了解可以腌制的内容:

The problem is that pickle needs to have some way to reassemble everything that you pickle. See here for a list of what can be pickled:

http://docs.python.org/library/pickle.html#what-c​​an-be-pickled-and-unpickled

酸洗my_func时，需要酸洗以下组件:

When pickling my_func, the following components need to be pickled:

  • my_decorator_class 的一个实例，称为 my_func.

  • An instance of my_decorator_class, called my_func.

这很好.Pickle 将存储类的名称并腌制其 __dict__ 内容.unpickling 时，它使用名称查找类，然后创建一个实例并填写 __dict__ 内容.但是，__dict__ 内容存在问题...

This is fine. Pickle will store the name of the class and pickle its __dict__ contents. When unpickling, it uses the name to find the class, then creates an instance and fills in the __dict__ contents. However, the __dict__ contents present a problem...

存储在 my_func.target 中的原始 my_func 的实例.

The instance of the original my_func that's stored in my_func.target.

这不太好.它是顶层的函数，通常可以腌制.Pickle 将存储函数的名称.然而，问题在于名称my_func"不再绑定到未修饰的函数，而是绑定到修饰的函数.这意味着 pickle 将无法查找未修饰的函数来重新创建对象.遗憾的是，pickle 无法知道它试图腌制的对象总是可以在名称 __main__.my_func 下找到.

This isn't so good. It's a function at the top-level, and normally these can be pickled. Pickle will store the name of the function. The problem, however, is that the name "my_func" is no longer bound to the undecorated function, it's bound to the decorated function. This means that pickle won't be able to look up the undecorated function to recreate the object. Sadly, pickle doesn't have any way to know that object it's trying to pickle can always be found under the name __main__.my_func.

你可以这样改变它，它会起作用:

You can change it like this and it will work:

import random
import multiprocessing
import functools

class my_decorator(object):
    def __init__(self, target):
        self.target = target
        try:
            functools.update_wrapper(self, target)
        except:
            pass

    def __call__(self, candidates, args):
        f = []
        for candidate in candidates:
            f.append(self.target([candidate], args)[0])
        return f

def old_my_func(candidates, args):
    f = []
    for c in candidates:
        f.append(sum(c))
    return f

my_func = my_decorator(old_my_func)

if __name__ == '__main__':
    candidates = [[random.randint(0, 9) for _ in range(5)] for _ in range(10)]
    pool = multiprocessing.Pool(processes=4)
    results = [pool.apply_async(my_func, ([c], {})) for c in candidates]
    pool.close()
    f = [r.get()[0] for r in results]
    print(f)

您已经观察到装饰器功能在类不起作用时起作用.我相信这是因为 functools.wraps 修改了装饰函数，使其具有它包装的函数的名称和其他属性.就 pickle 模块而言，它与普通的顶级函数没有区别，因此它通过存储其名称来腌制它.解压后，名称将绑定到装饰函数，因此一切正常.

You have observed that the decorator function works when the class does not. I believe this is because functools.wraps modifies the decorated function so that it has the name and other properties of the function it wraps. As far as the pickle module can tell, it is indistinguishable from a normal top-level function, so it pickles it by storing its name. Upon unpickling, the name is bound to the decorated function so everything works out.

相关文章