如何在导入的模块中使用 multiprocessing.Pool?
问题描述
我无法在这里实施建议:同时对两个列表应用两个函数.
I have not been able to implement the suggestion here: Applying two functions to two lists simultaneously.
我猜是因为该模块是由另一个模块导入的，因此我的 Windows 产生了多个 python 进程?
I guess it is because the module is imported by another module and thus my Windows spawns multiple python processes?
我的问题是:我如何在没有 if if __name__ == "__main__":
My question is: how can I use the code below without the if if __name__ == "__main__":
args_m = [(mortality_men, my_agents, graveyard, families, firms, year, agent) for agent in males]
args_f = [(mortality_women, fertility, year, families, my_agents, graveyard, firms, agent) for agent in females]
with mp.Pool(processes=(mp.cpu_count() - 1)) as p:
p.map_async(process_males, args_m)
p.map_async(process_females, args_f)
process_males 和 process_females 都是函数.args_m, args_f 是迭代器
Both process_males and process_females are fuctions.
args_m, args_f are iterators
另外，我不需要返回任何东西.代理是需要更新的类实例.
Also, I don't need to return anything. Agents are class instances that need updating.
解决方案
if __name__ == '__main__': 的想法是避免无限进程产生.
The idea of if __name__ == '__main__': is to avoid infinite process spawning.
当腌制在你的主脚本中定义的函数时，python 必须弄清楚你的主脚本的哪一部分是函数代码.它基本上会重新运行你的脚本.如果您创建 Pool 的代码在同一个脚本中并且不受if main"保护，那么通过尝试导入该函数，您将尝试启动另一个 Pool这将尝试启动另一个 Pool....
When pickling a function defined in your main script, python has to figure out what part of your main script is the function code. It will basically re run your script. If your code creating the Pool is in the same script and not protected by the "if main", then by trying to import the function, you will try to launch another Pool that will try to launch another Pool....
因此，您应该将函数定义与实际的主脚本分开:
Thus you should separate the function definitions from the actual main script:
from multiprocessing import Pool
# define test functions outside main
# so it can be imported withou launching
# new Pool
def test_func():
pass
if __name__ == '__main__':
with Pool(4) as p:
r = p.apply_async(test_func)
... do stuff
result = r.get()
相关文章