std::fstream 缓冲与手动缓冲(为什么手动缓冲能获得 10 倍的增益)?

2021-12-09 00:00:00 file buffer c++ c++11 fstream

我测试了两种写入配置:

I have tested two writing configurations:

  1. Fstream 缓冲:

  1. Fstream buffering:

// Initialization
const unsigned int length = 8192;
char buffer[length];
std::ofstream stream;
stream.rdbuf()->pubsetbuf(buffer, length);
stream.open("test.dat", std::ios::binary | std::ios::trunc)

// To write I use :
stream.write(reinterpret_cast<char*>(&x), sizeof(x));

  • 手动缓冲:

  • Manual buffering:

    // Initialization
const unsigned int length = 8192;
char buffer[length];
std::ofstream stream("test.dat", std::ios::binary | std::ios::trunc);

// Then I put manually the data in the buffer

// To write I use :
stream.write(buffer, length);

    • 我期待同样的结果...

    I expected the same result...

    但是我的手动缓冲将性能提高了 10 倍以写入 100MB 的文件，并且 fstream 缓冲与正常情况相比没有任何改变(没有重新定义缓冲区).

    But my manual buffering improve performance by a factor of 10 to write a file of 100MB, and the fstream buffering does not change anything compared to the normal situation (without redefining a buffer).

    有人对这种情况有解释吗?

    Does someone has an explanation of this situation ?

    这是新闻:刚刚在超级计算机上完成的基准测试(Linux 64 位架构，支持英特尔至强 8 核、Lustre 文件系统和...希望配置良好的编译器)(我没有解释 1kB 手动缓冲区共振"的原因......)

    EDIT : Here are the news : a benchmark just done on a supercomputer (linux 64-bit architecture, lasts intel Xeon 8-core, Lustre filesystem and ... hopefully well configured compilers) (and I don't explain the reason of the "resonance" for a 1kB manual buffer...)

    和 1024 B 的共振(如果有人对此有想法，我很感兴趣):

    EDIT 2 : And the resonance at 1024 B (if someone has an idea about that, I'm interested) :

    推荐答案

    这基本上是由于函数调用开销和间接性造成的.ofstream::write() 方法继承自 ostream.该函数未在 libstdc++ 中内联，这是第一个开销来源.然后 ostream::write() 必须调用 rdbuf()->sputn() 来做实际的写入，这是一个虚函数调用.

    This is basically due to function call overhead and indirection. The ofstream::write() method is inherited from ostream. That function is not inlined in libstdc++, which is the first source of overhead. Then ostream::write() has to call rdbuf()->sputn() to do the actual writing, which is a virtual function call.

    最重要的是，libstdc++ 将 sputn() 重定向到另一个虚函数 xsputn()，它添加了另一个虚函数调用.

    On top of that, libstdc++ redirects sputn() to another virtual function xsputn() which adds another virtual function call.

    如果您自己将字符放入缓冲区，则可以避免这种开销.

    If you put the characters into the buffer yourself, you can avoid that overhead.

    相关文章