多处理一个for循环?

2022-01-12 00:00:00 python multiprocessing

问题描述

我有一个数组(称为 data_inputs)，其中包含数百个天文图像文件的名称.然后对这些图像进行处理.我的代码有效，并且需要几秒钟来处理每个图像.但是，它一次只能做一个图像，因为我正在通过 for 循环运行数组:

I have an array (called data_inputs) containing the names of hundreds of astronomy images files. These images are then manipulated. My code works and takes a few seconds to process each image. However, it can only do one image at a time because I'm running the array through a for loop:

for name in data_inputs:
    sci=fits.open(name+'.fits')
    #image is manipulated

没有理由我必须先修改图像，所以是否可以利用我机器上的所有 4 个核心，每个核心在不同的图像上通过 for 循环运行?

There is no reason why I have to modify an image before any other, so is it possible to utilise all 4 cores on my machine with each core running through the for loop on a different image?

我已阅读有关 multiprocessing 模块的信息，但我不确定如何在我的情况下实现它.我热衷于让 multiprocessing 工作，因为最终我必须在 10,000 多张图像上运行它.

I've read about the multiprocessing module but I'm unsure how to implement it in my case. I'm keen to get multiprocessing to work because eventually I'll have to run this on 10,000+ images.

解决方案

你可以简单地使用 multiprocessing.Pool:

You can simply use multiprocessing.Pool:

from multiprocessing import Pool

def process_image(name):
    sci=fits.open('{}.fits'.format(name))
    <process>

if __name__ == '__main__':
    pool = Pool()                         # Create a multiprocessing Pool
    pool.map(process_image, data_inputs)  # process data_inputs iterable with pool

相关文章