Python多处理，传递包含信号量的对象引用

问题描述

我有这样的场景:我创建了一个包含信号量的类元素的对象.

import multiprocessing as mpr类元素(对象):def __init__(self):self.sem = mpr.Semaphore()自我.xyz = 33定义乐趣(ch):a = ch.recv()打印(一个[0])打印(a[1].xyz)a[1].xyz = 99打印(a[1].xyz)el = 元素()( pa , ch ) = mpr.Pipe()proc = mpr.Process(target=fun, args=( ch, ) )proc.start()pa.send([你好", el ])打印(el.xyz)proc.join()

此代码返回此错误:

 文件/usr/lib/python2.7/multiprocessing/forking.py"，第 51 行，在 assert_spawning 中'通过继承' % type(self).__name__RuntimeError:信号量对象只能通过继承在进程之间共享

但是如果我从 Element 的声明中删除信号量，代码可以工作，但是分配给 a[1].xyz 的值将会丢失.

现在我需要通过信号量和多处理同步大量对象.那么有一些方法可以在每个对象中设置一个信号量并只传递对主对象的引用吗?

import multiprocessing as mpr类元素(对象):def __init__(self):自我.xyz = 33定义乐趣(ch):a = ch.recv()打印(一个[0])打印(a[1].xyz)a[1].xyz = 99打印(a[1].xyz)el = 元素()( pa , ch ) = mpr.Pipe()proc = mpr.Process(target=fun, args=( ch, ) )proc.start()pa.send([你好", el ])打印(el.xyz)proc.join()

第二个版本不会产生任何错误，但是分配给a[1].xyz = 99的值会在主进程中丢失.

解决方案

我认为您不了解 multiprocessing 模块的工作原理.

当您通过管道发送某些内容时，它会在子流程中被腌制然后解除腌制.这意味着子进程实际上有一个原始对象的副本！这就是改变丢失"的原因.添加信号量不会改变任何东西.

如果您想要共享内存中的对象，您应该使用 multiprocessing.Value，即使这不处理任意类型.可能 multiprocessing.Manager 是您正在寻找的.

另一种方法是向提供修改对象的主进程发送响应.

I've a scenario like this: I've created an object of the class element containing a semaphore.

import multiprocessing as mpr

class Element(object):
    def __init__(self):
        self.sem = mpr.Semaphore()
        self.xyz = 33

def fun( ch ):
    a = ch.recv()
    print( a[0] )
    print( a[1].xyz )
    a[1].xyz = 99
    print( a[1].xyz )


el = Element()

( pa , ch ) = mpr.Pipe()
proc = mpr.Process(target=fun , args=( ch, ) )

proc.start()
pa.send( [ "Hallo" , el ])

print( el.xyz )

proc.join()

This code return this error:

  File "/usr/lib/python2.7/multiprocessing/forking.py", line 51, in assert_spawning
    ' through inheritance' % type(self).__name__
RuntimeError: Semaphore objects should only be shared between processes through inheritance

But if I remove the semaphore from the declaration of Element the code works, but the value assigned to a[1].xyz will be lost.

Now I need to synchronizes a big collection of object via semphore and multiprocessing. So there's some method for setting a semaphore in every object and passing only the reference to the main object?

import multiprocessing as mpr

class Element(object):
    def __init__(self):
        self.xyz = 33

def fun( ch ):
    a = ch.recv()
    print( a[0] )
    print( a[1].xyz )
    a[1].xyz = 99
    print( a[1].xyz )


el = Element()

( pa , ch ) = mpr.Pipe()
proc = mpr.Process(target=fun , args=( ch, ) )

proc.start()
pa.send( [ "Hallo" , el ])

print( el.xyz )

proc.join()

The second version dot't produce any error, but the value assigned to a[1].xyz = 99 will be lost in the main process.

解决方案

I don't think you understood how the multiprocessing module works.

When you send something through the pipe, it gets pickled and then unpickled in the subprocess. This means that the subprocess actually has a copy of the original object! That's why the change is "lost". Adding a semaphore wont change anything.

If you want an object in shared memory you should use multiprocessing.Value, even though this does not handle arbitrary types. Probably multiprocessing.Manager is what you are looking for.

An other way would be to send a response to the main process providing the modified object.