“块大小"multiprocessing.Pool.map 中的参数
问题描述
例如，如果我有一个带有 2 个处理器的池对象:
If I have a pool object with 2 processors for example:
p=multiprocessing.Pool(2)
我想遍历目录中的文件列表并使用 map 函数
and I want to iterate over a list of files on directory and use the map function
谁能解释一下这个函数的块大小是多少:
could someone explain what is the chunksize of this function:
p.map(func, iterable[, chunksize])
如果我将 chunksize 例如设置为 10，这是否意味着每 10 个文件都应该使用一个处理器进行处理?
If I set the chunksize for example to 10 does that means every 10 files should be processed with one processor?
解决方案
看Pool.map 的文档 看来您几乎是正确的: chunksize 参数将导致可迭代对象被拆分为大约该大小，并且每件作品都作为单独的任务提交.
Looking at the documentation for Pool.map it seems you're almost correct: the chunksize parameter will cause the iterable to be split into pieces of approximately that size, and each piece is submitted as a separate task.
所以在您的示例中，是的，map 将采用前 10 个(大约)，将其作为单个处理器的任务提交......然后接下来的 10 个将作为另一个任务提交，等等.请注意，这并不意味着这会使处理器每 10 个文件交替一次，很有可能处理器 #1 最终得到 1-10 和 11-20，而处理器 #2 得到 21-30 和 31-40.
So in your example, yes, map will take the first 10 (approximately), submit it as a task for a single processor... then the next 10 will be submitted as another task, and so on. Note that it doesn't mean that this will make the processors alternate every 10 files, it's quite possible that processor #1 ends up getting 1-10 AND 11-20, and processor #2 gets 21-30 and 31-40.
相关文章