Python多处理导致许多僵尸进程

2022-01-12 00:00:00 python multiprocessing

问题描述

我一直在使用一组工作人员来实现 python 的多处理库.我实现了以下代码

I have been implementing python's multiprocessing library using a pool of workers. I implemented the following code 

import main1
t1 = time.time()
p = Pool(cores) 
result = p.map(main1, client_list[client])
if result == []:
    return []
p.close()
p.join()
print "Time taken in performing request:: ", time.time()-t1
return shorted(result)

但是，在运行该进程一段时间后，我的应用程序有很多正在运行的后台进程.这是为我的应用执行 ps aux 后的快照

However, after running the process for a while, I get lot of running background processes of my app. Here is a snapshot after doing ps aux for my app

现在，我已经阅读了很多关于 stackoverflow 的类似问题，例如 如何杀死由多处理模块创建的僵尸进程? 这需要使用我已经实现的 .join() 并且我从这里学习了如何杀死所有这些进程 Python 多处理终止进程.但我想知道我的代码可能会出现什么问题.我无法在 main1 函数中共享我的所有代码，但我已将整个代码块放在 try catch 块中，以避免主代码中的错误可能导致僵尸进程的情况.

Now, I have read a lot of similar questions on stackoverflow like how to kill zombie processes created by multiprocessing module? which calls for using .join() which I have already implemented and I learned how to kill all these processes from here Python Multiprocessing Kill Processes. But I want to know what possibly could go wrong with my code. I won't able to share all of my code in the main1 function but I have put the entire code block in try catch block to avoid cases where an error in the main code could lead to zombie processes. 

def main1((param1, param2, param3)):
    try:
       resout.append(some_data) //resout in case of no error
    except:
        print traceback.format_exc()
        resout = []  //sending empty resout in case of error
    return resout

我对并行编程的概念还是很陌生，调试问题变得很棘手.任何帮助将不胜感激.

I'm still very new to the concept of parallel programming and debugging issues with it is turning out to be tricky.Any help will be greatly appreciated.

解决方案

通常最常见的问题是创建了池但是没有关闭.

Usually the most common problem is that the pool is created but it is not closed.

我知道保证池关闭的最好方法是使用 try/finally 子句:

The best way I know to guarantee that the pool is closed is to use a try/finally clause:

try:
    pool = Pool(ncores)
    pool.map(yourfunction, arguments)
finally:
    pool.close()
    pool.join()

如果您不想与 multiprocessing 作斗争，我编写了一个名为 parmap 的简单包，它封装了多处理，让我(可能还有你的)生活更轻松.

If you don't want to struggle with multiprocessing, I wrote a simple package named parmap that wraps multiprocessing to make my life (and potentially yours) easier.

pip install parmap

import parmap
parmap.map(yourfunction, arguments)

来自 parmap 使用部分:

From the parmap usage section:

  • 简单的并行示例:

  • Simple parallel example:

import parmap
y1 = [myfunction(x, argument1, argument2) for x in mylist]
y2 = parmap.map(myfunction, mylist, argument1, argument2)
y1 == y2

  • 遍历元组列表:

  • Iterating over a list of tuples:

    # You want to do:
z = [myfunction(x, y, argument1, argument2) for (x,y) in mylist]
z = parmap.starmap(myfunction, mylist, argument1, argument2)


# You want to do:
listx = [1, 2, 3, 4, 5, 6]
listy = [2, 3, 4, 5, 6, 7]
param = 3.14
param2 = 42
listz = []
for (x, y) in zip(listx, listy):
    listz.append(myfunction(x, y, param1, param2))
# In parallel:
listz = parmap.starmap(myfunction, zip(listx, listy), param1, param2)

    • 相关文章