Python3.8下的模块多处理出错

问题描述

我有一个脚本，直到今天都可以很好地进行多处理。为了重现这个问题，我简化了我并行化的函数，如下所示：

    from multiprocessing import Process, Queue
    import random

    def rand_num():
        num = random.random()
        print(num)

    if __name__ == "__main__":
        queue = Queue()

        processes = [Process(target=rand_num, args=()) for x in range(4)]

        for p in processes:
            p.start()

        for p in processes:
            p.join()

这将呈现完全相同的错误消息(重复4次，为了可读性，我省略了重复)：

    Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
    File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
    File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
    File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
    File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 262, in run_path
    code, fname = _get_code_from_file(run_name, path_name)
    File "/usr/local/Cellar/python@3.8/3.8.1/Frameworks/Python.framework/Versions/3.8/lib/python3.8/runpy.py", line 232, in _get_code_from_file
    with io.open_code(fname) as f:
    FileNotFoundError: [Errno 2] No such file or directory: '/Users/myUserName/<stdin>'

我不知道从哪里开始调试此错误。我在mac os Catalina(自制软件安装)下运行python3.8。请帮帮忙。

---

解决方案

我在从Python3.7升级到3.8时遇到了同样的问题。 尤其是现在在OSX 10.15.6上运行3.8.6，由pyenv安装的Python。

黑暗航行者的建议帮助解决了这个问题，但它并不那么明显，所以让我在这里重新表述一下：

MacOS上的Python3.8现在默认使用spawn而不是fork作为新进程的启动方法。尝试

multiprocessing.set_start_method("fork")

很明显，产卵的行为是错误的，如下面的简单示例所示：

import multiprocessing

def parallel_function(x):
    print("Function called with", x)

def test_pool():
    print("Running test_pool")
    with multiprocessing.Pool(4) as pool:
        pool.map(parallel_function, range(10))

print("Starting the test")
test_pool()

这将产生以下输出：

Starting the test
Running test_pool
Starting the test
Running test_pool
Starting the test
Running test_pool
Starting the test
Running test_pool
Starting the test
Running test_pool
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 116, in spawn_main
    exitcode = _main(fd, parent_sentinel)
  File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 125, in _main
    prepare(preparation_data)
  File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 236, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/Users/karel/.pyenv/versions/3.8.6/lib/python3.8/multiprocessing/spawn.py", line 287, in _fixup_main_from_path

因此池不会正确创建工作进程，而是尝试在每个派生的进程中运行整个脚本。