在 Pandas 中使用多处理读取 csv 文件的最简单方法

2022-01-12 00:00:00 python pandas csv multiprocessing

问题描述

这是我的问题.
带有一堆 .csv 文件(或其他文件).Pandas 是一种读取它们并保存为 Dataframe 格式的简单方法.但是当文件量很大时，我想通过多处理读取文件以节省一些时间.

Here is my question.
With bunch of .csv files(or other files). Pandas is an easy way to read them and save into Dataframe format. But when the amount of files was huge, I want to read the files with multiprocessing to save some time.

我手动将文件分成不同的路径.单独使用:

I manually divide the files into different path. Using severally:

os.chdir("./task_1")
files = os.listdir('.')
files.sort()
for file in files:
    filename,extname = os.path.splitext(file)
    if extname == '.csv':
        f = pd.read_csv(file)
        df = (f.VALUE.as_matrix()).reshape(75,90)

然后将它们组合起来.

如何使用 pool 运行它们来解决我的问题?
任何建议将不胜感激！

How to run them with pool to achieve my problem?
Any advice would be appreciated!

解决方案

使用Pool:

import os
import pandas as pd 
from multiprocessing import Pool

# wrap your csv importer in a function that can be mapped
def read_csv(filename):
    'converts a filename to a pandas dataframe'
    return pd.read_csv(filename)


def main():

    # get a list of file names
    files = os.listdir('.')
    file_list = [filename for filename in files if filename.split('.')[1]=='csv']

    # set up your pool
    with Pool(processes=8) as pool: # or whatever your hardware can support

        # have your pool map the file names to dataframes
        df_list = pool.map(read_csv, file_list)

        # reduce the list of dataframes to a single dataframe
        combined_df = pd.concat(df_list, ignore_index=True)

if __name__ == '__main__':
    main()

相关文章