python multiprocessing vs threading for cpu bound work on windows and linux

2022-01-12 00:00:00 python multiprocessing

问题描述

所以我敲了一些测试代码，看看多处理模块在 cpu 绑定工作上与线程相比如何扩展.在 linux 上，我得到了预期的性能提升:

So I knocked up some test code to see how the multiprocessing module would scale on cpu bound work compared to threading. On linux I get the performance increase that I'd expect:

linux (dual quad core xeon):
serialrun took 1192.319 ms
parallelrun took 346.727 ms
threadedrun took 2108.172 ms

我的双核 macbook pro 显示相同的行为:

My dual core macbook pro shows the same behavior:

osx (dual core macbook pro)
serialrun took 2026.995 ms
parallelrun took 1288.723 ms
threadedrun took 5314.822 ms

然后我在一台windows机器上试了一下，得到了一些非常不同的结果.

I then went and tried it on a windows machine and got some very different results.

windows (i7 920):
serialrun took 1043.000 ms
parallelrun took 3237.000 ms
threadedrun took 2343.000 ms

为什么，为什么，Windows 上的多处理方法这么慢?

Why oh why, is the multiprocessing approach so much slower on windows?

这是测试代码:

#!/usr/bin/env python

import multiprocessing
import threading
import time

def print_timing(func):
    def wrapper(*arg):
        t1 = time.time()
        res = func(*arg)
        t2 = time.time()
        print '%s took %0.3f ms' % (func.func_name, (t2-t1)*1000.0)
        return res
    return wrapper


def counter():
    for i in xrange(1000000):
        pass

@print_timing
def serialrun(x):
    for i in xrange(x):
        counter()

@print_timing
def parallelrun(x):
    proclist = []
    for i in xrange(x):
        p = multiprocessing.Process(target=counter)
        proclist.append(p)
        p.start()

    for i in proclist:
        i.join()

@print_timing
def threadedrun(x):
    threadlist = []
    for i in xrange(x):
        t = threading.Thread(target=counter)
        threadlist.append(t)
        t.start()

    for i in threadlist:
        i.join()

def main():
    serialrun(50)
    parallelrun(50)
    threadedrun(50)

if __name__ == '__main__':
    main()

解决方案

进程在 UNIX 变体下更加轻量级.Windows 进程很繁重，需要更多时间才能启动.线程是在 Windows 上进行多处理的推荐方式.

Processes are much more lightweight under UNIX variants. Windows processes are heavy and take much more time to start up. Threads are the recommended way of doing multiprocessing on windows.

相关文章