使用 numpy 进行多处理使 Python 在 OSX 上意外退出

2022-01-12 00:00:00 python numpy macos multiprocessing

问题描述

我遇到了一个问题，即 Python 在使用 numpy 运行多处理时意外退出.我已经隔离了这个问题，因此我现在可以确认在运行以下代码时多处理工作正常:

I've run into a problem, where Python quits unexpectedly, when running multiprocessing with numpy. I've isolated the problem, so that I can now confirm that the multiprocessing works perfect when running the code stated below:

import numpy as np
from multiprocessing import Pool, Process
import time
import cPickle as p

def test(args):
    x,i = args
    if i == 2:
        time.sleep(4)
    arr = np.dot(x.T,x)
    print i

if __name__ == '__main__':
    x = np.random.random(size=((2000,500)))
    evaluations = [(x,i) for i in range(5)]
    p = Pool()
    p.map_async(test,evaluations)
    p.close()
    p.join()

当我尝试评估下面的代码时会出现问题.这使得 Python 意外退出:

The problem occurs when I try to evaluate the code below. This makes Python quit unexpectedly:

import numpy as np
from multiprocessing import Pool, Process
import time
import cPickle as p

def test(args):
    x,i = args
    if i == 2:
        time.sleep(4)
    arr = np.dot(x.T,x)
    print i

if __name__ == '__main__':
    x = np.random.random(size=((2000,500)))
    test((x,4)) # Added code
    evaluations = [(x,i) for i in range(5)]
    p = Pool()
    p.map_async(test,evaluations)
    p.close()
    p.join()

请帮助某人.我愿意接受所有建议.谢谢.注意:我试过两台不同的机器，都出现同样的问题.

Please help someone. I'm open to all suggestions. Thanks. Note: I have tried two different machines and the same problem occurs.

解决方案

我找到了解决问题的方法.在初始化多处理实例之前将 Numpy 与 BLAS 一起使用时会出现此问题.我的解决方法是将 Numpy 代码(运行 BLAS)放入单个进程中，然后运行多处理实例.这不是一种好的编码风格，但它确实有效.请参见下面的示例:

I figured out a workaround to the problem. The problem occurs when Numpy is used together with BLAS before initializing a multiprocessing instance. My workaround is simply to put the Numpy code (running BLAS) into a single process and then running the multiprocessing instances afterwards. This is not a good coding style, but it works. See example below:

以下将失败 - Python 将退出:

Following will fail - Python will quit:

import numpy as np
from multiprocessing import Pool, Process

def test(x):
    arr = np.dot(x.T,x) # On large matrices, this calc will use BLAS.

if __name__ == '__main__':
    x = np.random.random(size=((2000,500))) # Random matrix
    test(x)
    evaluations = [x for _ in range(5)]
    p = Pool()
    p.map_async(test,evaluations) # This is where Python will quit, because of the prior use of BLAS.
    p.close()
    p.join()

以下会成功:

import numpy as np
from multiprocessing import Pool, Process

def test(x):
    arr = np.dot(x.T,x) # On large matrices, this calc will use BLAS.

if __name__ == '__main__':
    x = np.random.random(size=((2000,500))) # Random matrix
    p = Process(target = test,args = (x,))
    p.start()
    p.join()
    evaluations = [x for _ in range(5)]
    p = Pool()
    p.map_async(test,evaluations)
    p.close()
    p.join()

相关文章