无法腌制 <type 'instancemethod'>当使用多处理 Pool.map()

2022-01-12 00:00:00 python multithreading multiprocessing pickle pool

问题描述

我正在尝试使用 multiprocessing 的 Pool.map() 函数同时划分工作.当我使用以下代码时，它工作正常:

I'm trying to use multiprocessing's Pool.map() function to divide out work simultaneously. When I use the following code, it works fine:

import multiprocessing

def f(x):
    return x*x

def go():
    pool = multiprocessing.Pool(processes=4)        
    print pool.map(f, range(10))


if __name__== '__main__' :
    go()

但是，当我在更面向对象的方法中使用它时，它就不起作用了.它给出的错误信息是:

However, when I use it in a more object-oriented approach, it doesn't work. The error message it gives is:

PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
__builtin__.instancemethod failed

当以下是我的主程序时会发生这种情况:

This occurs when the following is my main program:

import someClass

if __name__== '__main__' :
    sc = someClass.someClass()
    sc.go()

以下是我的 someClass 类:

import multiprocessing

class someClass(object):
    def __init__(self):
        pass

    def f(self, x):
        return x*x

    def go(self):
        pool = multiprocessing.Pool(processes=4)       
        print pool.map(self.f, range(10))

任何人都知道问题可能是什么，或者解决它的简单方法?

Anyone know what the problem could be, or an easy way around it?

解决方案

问题是多处理必须腌制东西以将它们吊在进程之间，并且绑定的方法是不可腌制的.解决方法(无论您是否认为它容易"；-)是将基础结构添加到您的程序中以允许对此类方法进行腌制，并使用 copy_reg 标准库方法.

The problem is that multiprocessing must pickle things to sling them among processes, and bound methods are not picklable. The workaround (whether you consider it "easy" or not;-) is to add the infrastructure to your program to allow such methods to be pickled, registering it with the copy_reg standard library method.

例如，Steven Bethard 对 此线程(接近线程的末尾)显示了一种完全可行的方法，允许通过 copy_reg 进行方法酸洗/取消酸洗.

For example, Steven Bethard's contribution to this thread (towards the end of the thread) shows one perfectly workable approach to allow method pickling/unpickling via copy_reg.

相关文章