Python 多处理的池进程限制

2022-01-12 00:00:00 python multiprocessing cpu-cores

问题描述

在使用多处理模块的 Pool 对象时，进程数是否受 CPU 核数的限制?例如.如果我有 4 个核心，即使我创建一个包含 8 个进程的池，一次也只能运行 4 个?

In using the Pool object from the multiprocessing module, is the number of processes limited by the number of CPU cores? E.g. if I have 4 cores, even if I create a Pool with 8 processes, only 4 will be running at one time?

解决方案

你可以要求尽可能多的进程.任何可能存在的限制都将由您的操作系统施加，而不是由 multiprocessing 施加.例如，

You can ask for as many processes as you like. Any limit that may exist will be imposed by your operating system, not by multiprocessing. For example,

 p = multiprocessing.Pool(1000000)

在任何机器上都可能遭受丑陋的死亡.当我输入这个时，我正在我的盒子上尝试它，并且操作系统正在将我的磁盘磨成灰尘，疯狂地换出 RAM - 最后在它创建了大约 3000 个进程后将其杀死；-)

is likely to suffer an ugly death on any machine. I'm trying it on my box as I type this, and the OS is grinding my disk to dust swapping out RAM madly - finally killed it after it had created about 3000 processes ;-)

至于一次"运行多少个，Python 没有发言权.这取决于:

As to how many will run "at one time", Python has no say in that. It depends on:

  1. 您有多少硬件能够同时运行；并且，
  2. 您的操作系统如何决定将硬件资源分配给您计算机上当前正在运行的所有进程.
  1. How many your hardware is capable of running simultaneously; and,
  2. How your operating system decides to give hardware resources to all the processes on your machine currently running.

对于 CPU 密集型任务，创建比运行它们的内核更多的 Pool 进程没有意义.如果您也尝试将您的机器用于其他事情，那么您应该创建比内核更少的进程.

For CPU-bound tasks, it doesn't make sense to create more Pool processes than you have cores to run them on. If you're trying to use your machine for other things too, then you should create fewer processes than cores.

对于 I/O 密集型任务，可能创建比核心更多的 Pool 进程，因为这些进程可能会花费大部分时间阻塞(等待 I/O 完成).

For I/O-bound tasks, it may make sense to create a quite a few more Pool processes than cores, since the processes will probably spend most their time blocked (waiting for I/O to complete).

相关文章