Python - 在 FTP 中按块上传内存文件(由 API 调用生成)

2022-01-09 00:00:00 python sftp paramiko ftp ftplib

问题描述

我需要能够在 Python 中通过 FTP 和 SFTP 上传文件，但有一些不太常见的限制.

I need to be able to upload a file through FTP and SFTP in Python but with some not so usual constraints.

  1. 文件不得写入磁盘.

  1. File MUST NOT be written in disk.

文件的生成方式是调用 API 并将 JSON 格式的响应写入文件.

The file how it is generated is by calling an API and writing the response which is in JSON to the file.

API 有多次调用.无法通过一次 API 调用来检索整个结果.

There are multiple calls to the API. It is not possible to retrieve the whole result in one single call of the API.

我无法将完整的结果存储在字符串变量中，方法是执行所需的多次调用并附加到每个调用中，直到我将整个文件存储在内存中.文件可能很大并且存在内存资源限制.应该发送每个块并释放内存.

I can not store in a string variable the full result by doing the multiple calls needed and appending in each call until I have the whole file in memory. File could be huge and there is a memory resource constraint. Each chunk should be sent and memory deallocated.

所以这里有一些我想要的示例代码:

So here some sample code of what I would like to:

def chunks_generator():
    range_list = range(0, 4000, 100)
    for i in range_list:
        data_chunk = requests.get(url=someurl, url_parameters={'offset':i, 'limit':100})
        yield str(data_chunk)

def upload_file():
    chunks_generator = chunks_generator()
    for chunk in chunks_generator:
        data_chunk= chunk
        chunk_io = io.BytesIO(data_chunk)
        ftp = FTP(self.host)
        ftp.login(user=self.username, passwd=self.password)
        ftp.cwd(self.remote_path)
        ftp.storbinary("STOR " + "myfilename.json", chunk_io)

我只想要一个附加了所有块的文件.我已经拥有并且正在工作的是，如果我将整个文件保存在内存中并像这样立即发送它:

I want only one file with all the chunks appended. What I have already and works is if I have the whole file in memory and send it at once like this:

string_io = io.BytesIO(all_chunks_together_in_one_string)
ftp = FTP(self.host)
ftp.login(user=self.username, passwd=self.password)
ftp.cwd(self.remote_path)
ftp.storbinary("STOR " + "myfilename.json", string_io )

奖金

我在 ftplib 中需要它，但在 Paramiko 中也需要它用于 SFTP.如果有任何其他库可以更好地工作，我是开放的.

I need this in ftplib but will need it in Paramiko as well for SFTP. If there are any other libraries that this would work better I am open.

如果我需要压缩文件怎么办?我可以压缩每个块并一次发送 zip-chunked 块吗?

How about if I need to zip the file? Can I zip each chunk and send the zip-chunked chunk at a time?

解决方案

可以实现类文件类，调用.read(blocksize)方法从requests 对象.

You can implement file-like class that upon calling .read(blocksize) method retrieves data from requests object.

类似这样的东西(未经测试):

Something like this (untested):

class ChunksGenerator:
    i = 0
    requests = None

    def __init__(self, requests)
        self.requests = requests

    def read(self, blocksize):
        # TODO: somehow detect end-of-file and return false in that case
        buf = requests.get(
                  url=someurl, url_parameters={'offset':self.i, 'limit':blocksize})
        self.i += blocksize
        return buf

generator = ChunksGenerator(requests)
ftp.storbinary("STOR " + "myfilename.json", generator)

<小时>

使用 Paramiko，您可以使用与 SFTPClient.putfo 方法.

相关文章