如何使用原生 sftp 函数和 PHP 中的 fwrite 提高文件上传的性能

2022-01-09 00:00:00 sftp upload php fwrite ssh2-sftp

您好，我正在使用以下代码将一个大文件 (500MB) 上传到 sftp 服务器.

Hi I am using the following code to upload a huge file (500MB) to a sftp server.

<?php

$connection = ssh2_connect($this->host, $this->port, null);
$sftp = ssh2_sftp($connection);

$connection_string = ((int) $sftp) . $remotePath . $remoteFilename;
$stream = fopen('ssh2.sftp://' . $connection_string, 'w');
$source = fopen($localFilepath, 'r');

if (!$stream) {
    throw new Exception('Could not create file: ' . $connection_string);
}

while (!feof($source)) {
    // Chunk size 32 MB
    if (fwrite($stream, fread($source, 33554432)) === false) {
        throw new Exception('Could not send data: ' . $connection_string);
    }
}

fclose($source);
fclose($stream);

但上传速度非常慢.代码在 Google Cloud Run 上运行.上传速度约为 8 MiB/s.

But the upload is very slow. The code is running on Google Cloud Run. The upload speed is around 8 MiB/s.

我也尝试通过 shell_exec 使用 lftp，但由于 Cloud Run，这会导致更多问题.

I also tried to use lftp via shell_exec but this lead to even more issues due to Cloud Run.

上行链路不是问题，因为我可以通过 CURL post 发送文件而没有任何问题.

The uplink can't be the problem as I can send files via CURL post without any issues.

有人可以帮忙吗?

非常感谢和最好的，intxcc

Many thanks and best, intxcc

推荐答案

问题是即使 32MB 被读取然后写入 sftp 流，fwrite 也会以不同的大小分块.我想只有几 KB.

The issue is that even though 32MB are read and then written to the sftp stream, fwrite will chunk at a different size. I think just a few KB.

对于文件系统(这是 fwrite 的常见情况)，这很好，但由于 fwrite 到远程服务器而导致高延迟.

For filesystems (which is the usual case with fwrite) this is fine, but not with high latency due to fwriting to a remote server.

所以解决方法是增加sftp流的chunk大小

So the solution is to increase the chunk size of the sftp stream with

stream_set_chunk_size($stream, 1024 * 1024);

所以最终的工作代码是:

So the final working code is:

<?php

$connection = ssh2_connect($this->host, $this->port, null);
$sftp = ssh2_sftp($connection);

$connection_string = ((int) $sftp) . $remotePath . $remoteFilename;
$stream = fopen('ssh2.sftp://' . $connection_string, 'w');
$source = fopen($localFilepath, 'r');

// Stream chunk size 1 MB
stream_set_chunk_size($stream, 1024 * 1024);

if (!$stream) {
    throw new Exception('Could not create file: ' . $connection_string);
}

while (!feof($source)) {
    // Chunk size 32 MB
    if (fwrite($stream, fread($source, 33554432)) === false) {
        throw new Exception('Could not send data: ' . $connection_string);
    }
}

fclose($source);
fclose($stream);

希望这有助于下一个白发的人试图弄清楚这一点；)

Hope this helps the next person that is getting gray hair trying to figure that out ;)

相关文章