multiprocessing.Manager 嵌套共享对象不适用于队列

问题描述

Python docs of the multiprocessing module state:

Changed in version 3.6: Shared objects are capable of being nested. For example, a shared container object such as a shared list can contain other shared objects which will all be managed and synchronized by the SyncManager.

This does work with list and dict. However, if I try to create a shared Queue inside a shared dict, I get an error:

>>> from multiprocessing import Manager
>>> m = Manager()
>>> d = m.dict()
>>> d['a'] = m.list()
>>> d['b'] = m.dict()
>>> d['c'] = m.Queue()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 2, in __setitem__
  File "/usr/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod
    raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError: 
---------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/managers.py", line 228, in serve_client
    request = recv()
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 251, in recv
    return _ForkingPickler.loads(buf.getbuffer())
  File "/usr/lib/python3.6/multiprocessing/managers.py", line 881, in RebuildProxy
    return func(token, serializer, incref=incref, **kwds)
TypeError: AutoProxy() got an unexpected keyword argument 'manager_owned'
---------------------------------------------------------------------------

Seems like https://hg.python.org/cpython/rev/39e7307f9aee is the changeset which introduced nested shared objects.

解决方案

The error is caused by AutoProxy currently not handling all BaseProxy arguments. There's a pull request which has not been merged yet. You either need to monkey-patch AutoProxy, or you look into multiprocessing.managers.py and apply the changes in the patch here directly to your source code.

It's really important to fix both lines in the patch to prevent a memory leak in the server process. The manager_owned-flag is used to let the BaseProxy code know, when to skip a reference increment for a proxy the manager owns himself (through nesting).