在 Python 中将多个参数传递给 pool.map() 函数

2022-01-12 00:00:00 python multiprocessing pool map-function

问题描述

我需要一些方法来使用 pool.map() 中接受多个参数的函数.根据我的理解， pool.map() 的目标函数只能有一个可迭代的参数，但是有没有一种方法可以传递其他参数?在这种情况下，我需要传入一些配置变量，例如我的 Lock() 和日志信息到目标函数.

I need some way to use a function within pool.map() that accepts more than one parameter. As per my understanding, the target function of pool.map() can only have one iterable as a parameter but is there a way that I can pass other parameters in as well? In this case, I need to pass in a few configuration variables, like my Lock() and logging information to the target function.

我试图做一些研究，我认为我可以使用部分函数来让它工作?但是我不完全理解这些是如何工作的.任何帮助将不胜感激！这是我想做的一个简单示例:

I have tried to do some research and I think that I may be able to use partial functions to get it to work? However I don't fully understand how these work. Any help would be greatly appreciated! Here is a simple example of what I want to do:

def target(items, lock):
    for item in items:
        # Do cool stuff
        if (... some condition here ...):
            lock.acquire()
            # Write to stdout or logfile, etc.
            lock.release()

def main():
    iterable = [1, 2, 3, 4, 5]
    pool = multiprocessing.Pool()
    pool.map(target(PASS PARAMS HERE), iterable)
    pool.close()
    pool.join()

解决方案

你可以使用functools.partial 为此(正如您所怀疑的那样):

You can use functools.partial for this (as you suspected):

from functools import partial

def target(lock, iterable_item):
    for item in iterable_item:
        # Do cool stuff
        if (... some condition here ...):
            lock.acquire()
            # Write to stdout or logfile, etc.
            lock.release()

def main():
    iterable = [1, 2, 3, 4, 5]
    pool = multiprocessing.Pool()
    l = multiprocessing.Lock()
    func = partial(target, l)
    pool.map(func, iterable)
    pool.close()
    pool.join()

例子:

def f(a, b, c):
    print("{} {} {}".format(a, b, c))

def main():
    iterable = [1, 2, 3, 4, 5]
    pool = multiprocessing.Pool()
    a = "hi"
    b = "there"
    func = partial(f, a, b)
    pool.map(func, iterable)
    pool.close()
    pool.join()

if __name__ == "__main__":
    main()

输出:

hi there 1
hi there 2
hi there 3
hi there 4
hi there 5

相关文章