Python多处理附加列表

2022-01-12 00:00:00 python multiprocessing variables shared

问题描述

关于使用 Multiprocessing.Pool() 在多个进程之间共享变量的问题.

Have a quick question about a shared variable between multiple processes using Multiprocessing.Pool().

如果我从多个进程中更新全局列表，我会遇到任何问题吗?IE.如果两个进程同时尝试更新列表.

Will I run in to any issues if I am updating a global list from within multiple processes? I.e. if two of the processes were to try to update the list at the same time.

我看过关于在类似事情上使用锁的文档，但我想知道是否有必要.

I have seen documentation about using a Lock for similar things but I was wondering if it was necessary.

我共享这个变量的方式是在我的回调函数中使用一个全局变量，'success' 在目标函数完成后，我将所有成功的操作附加到:

The way I am sharing this variable is by using a global variable in my callback function, 'successes' in which i append all of the successful actions to after the target function has completed:

TOTAL_SUCCESSES = []

def func(inputs):
    successes = []

    for input in inputs:
        result = #something with return code
        if result == 0:
            successes.append(input)
    return successes

def callback(successes):
    global TOTAL_SUCCESSES

    for entry in successes:
        TOTAL_SUCCESSES.append(entry)

def main():     
    pool = mp.Pool()
    for entry in myInputs:
         pool.apply_async(func, args=(entry,),callback=callback)

为任何语法错误道歉，很快就写出来了，但是程序正在运行，只是想知道如果我有问题我是否添加了共享变量.

Apologize for any syntax errors, wrote this up quickly but the program is working just wondering if I add the shared variable if I will have issues.

提前致谢！

解决方案

使用您当前的代码，您实际上并没有在进程之间共享 CURRENT_SUCCESSES.callback 在主进程中的结果处理线程中执行.只有一个结果处理线程，因此每个 callback 将一次运行一个，而不是同时运行.所以你写的代码是进程/线程安全的.

With your current code, you're not actually sharing CURRENT_SUCCESSES between processes. callback is executed in the main process, in a result handling thread. There is only one result handling thread, so each callback will be run one at a time, not concurrently. So your code as written is process/thread safe.

但是，您忘记从 func 中返回 success，这是您想要修复的.

However, you are forgetting to return successes from func, which you'll want to fix.

此外，使用 map 可以更简洁地编写:

Also, this could be much more succinctly written using map:

def func(inputs):
    successes = []

    for input in inputs:
        result = #something with return code
        if result == 0:
            successes.append(input)
    return successes

def main():     
    pool = mp.Pool()
    total_successes = pool.map(func, myInputs) # Returns a list of lists
    # Flatten the list of lists
    total_successes = [ent for sublist in total_successes for ent in sublist]

相关文章