无法腌制 <type 'instancemethod'>使用 python 的多处理 Pool.apply_async()

2022-01-12 00:00:00 python multiprocessing pool

问题描述

我想运行这样的东西:

from multiprocessing  import Pool
import time
import random

class Controler(object):
    def __init__(self):
        nProcess = 10
        pages = 10
        self.__result = []
        self.manageWork(nProcess,pages)

def BarcodeSearcher(x):
    return x*x

def resultCollector(self,result):
    self.__result.append(result)

def manageWork(self,nProcess,pages):
    pool = Pool(processes=nProcess)
    for pag in range(pages):
        pool.apply_async(self.BarcodeSearcher, args = (pag, ), callback = self.resultCollector)

    print self.__result

if __name__ == '__main__':
    Controler()

但代码导致错误:

   Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:Python26lib	hreading.py", line 522, in __bootstrap_inner
    self.run()
  File "C:Python26lib	hreading.py", line 477, in run
    self.__target(*self.__args, **self.__kwargs)
  File "C:python26libmultiprocessingpool.py", line 225, in _handle_tasks
    put(task)
PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

我看过帖子(post1 , post2) 来解决我的问题.我在第二篇文章中寻找类似 Mike McKerns 的解决方案，但没有使用悲情.

I've seen the posts (post1 , post2) to solve my problem. I'm looking for something like Mike McKerns solution in the second post but without using pathos.

解决方案

这工作，使用 copy_reg，正如 Alex Martelli 在您提供的第一个链接中所建议的那样:

This works, using copy_reg, as suggested by Alex Martelli in the first link you provided:

import copy_reg
import types
import multiprocessing


def _pickle_method(m):
    if m.im_self is None:
        return getattr, (m.im_class, m.im_func.func_name)
    else:
        return getattr, (m.im_self, m.im_func.func_name)

copy_reg.pickle(types.MethodType, _pickle_method)


class Controler(object):
    def __init__(self):
        nProcess = 10
        pages = 10
        self.__result = []
        self.manageWork(nProcess, pages)

    def BarcodeSearcher(self, x):
        return x*x

    def resultCollector(self, result):
        self.__result.append(result)

    def manageWork(self, nProcess, pages):
        pool = multiprocessing.Pool(processes=nProcess)
        for pag in range(pages):
            pool.apply_async(self.BarcodeSearcher, args=(pag,),
                             callback=self.resultCollector)
        pool.close()
        pool.join()

        print(self.__result)

if __name__ == '__main__':
    Controler()

相关文章