在 Python 中读取大文件的惰性方法?

2022-01-30 00:00:00 python generator file-io

问题描述

我有一个非常大的 4GB 文件，当我尝试读取它时，我的计算机挂起.所以我想一块一块地读取它，在处理完每一块后将处理后的块存储到另一个文件中并读取下一块.

I have a very big file 4GB and when I try to read it my computer hangs. So I want to read it piece by piece and after processing each piece store the processed piece into another file and read next piece.

有什么方法可以yield这些片段吗?

Is there any method to yield these pieces ?

我希望有一个懒惰的方法.

解决方案

要写一个惰性函数，只需使用 yield:

To write a lazy function, just use yield:

def read_in_chunks(file_object, chunk_size=1024):
    """Lazy function (generator) to read a file piece by piece.
    Default chunk size: 1k."""
    while True:
        data = file_object.read(chunk_size)
        if not data:
            break
        yield data


with open('really_big_file.dat') as f:
    for piece in read_in_chunks(f):
        process_data(piece)

<小时>

另一种选择是使用 iter和一个辅助函数:

Another option would be to use iter and a helper function:

f = open('really_big_file.dat')
def read1k():
    return f.read(1024)

for piece in iter(read1k, ''):
    process_data(piece)

<小时>

如果文件是基于行的，则文件对象已经是行的惰性生成器:

If the file is line-based, the file object is already a lazy generator of lines:

for line in open('really_big_file.dat'):
    process_data(line)

相关文章