多处理 - 共享数组

2022-01-12 00:00:00 python multiprocessing shared

问题描述

所以我试图在 python 中实现多处理，我希望有一个由 4-5 个进程组成的池并行运行一个方法.这样做的目的是运行总共一千次 Monte 模拟(每个进程 250-200 次模拟)而不是运行 1000 次.我希望每个进程在处理完一个模拟的结果，写入结果并释放锁.所以这应该是一个三步过程:

So I'm trying to implement multiprocessing in python where I wish to have a Pool of 4-5 processes running a method in parallel. The purpose of this is to run a total of thousand Monte simulations (250-200 simulations per process) instead of running 1000. I want each process to write to a common shared array by acquiring a lock on it as soon as its done processing the result for one simulation, writing the result and releasing the lock. So it should be a three step process :

  1. 获取锁
  2. 写入结果
  3. 为等待写入数组的其他进程释放锁.

每次我将数组传递给进程时，每个进程都会创建一个我不想要的数组副本，因为我想要一个公共数组.任何人都可以通过提供示例代码来帮助我吗?

Everytime I pass the array to the processes each process creates a copy of that array which I donot want as I want a common array. Can anyone help me with this by providing sample code?

解决方案

由于您只是将状态从子进程返回到父进程，因此使用共享数组和显式锁是多余的.你可以使用 Pool.map 或 Pool.starmap 来完成你所需要的.例如:

Since you're only returning state from the child process to the parent process, then using a shared array and explicity locks is overkill. You can use Pool.map or Pool.starmap to accomplish exactly what you need. For example:

from multiprocessing import Pool

class Adder:
    """I'm using this class in place of a monte carlo simulator"""

    def add(self, a, b):
        return a + b

def setup(x, y, z):
    """Sets up the worker processes of the pool. 
    Here, x, y, and z would be your global settings. They are only included
    as an example of how to pass args to setup. In this program they would
    be "some arg", "another" and 2
    """
    global adder
    adder = Adder()

def job(a, b):
    """wrapper function to start the job in the child process"""
    return adder.add(a, b)

if __name__ == "__main__":   
    args = list(zip(range(10), range(10, 20)))
    # args == [(0, 10), (1, 11), ..., (8, 18), (9, 19)]

    with Pool(initializer=setup, initargs=["some arg", "another", 2]) as pool:
        # runs jobs in parallel and returns when all are complete
        results = pool.starmap(job, args)

    print(results) # prints [10, 12, ..., 26, 28]

相关文章