如何让 IOStream 性能更好?

2021-12-20 00:00:00 optimization iostream c++ c++-standard-library c++-faq

大多数学习 C 的 C++ 用户更喜欢使用 printf/scanf 系列函数，即使他们在使用 C++ 编码.

虽然我承认我发现界面更好(尤其是类似 POSIX 的格式和本地化)，但似乎压倒性的关注是性能.

看看这个问题:

<块引用>

如何加快逐行读取文件的速度

似乎最好的答案是使用 fscanf 并且 C++ ifstream 始终慢 2-3 倍.

我认为如果我们可以编译一个技巧"存储库来提高 IOStreams 性能，哪些有效，哪些无效.

要考虑的要点

缓冲 (rdbuf()->pubsetbuf(buffer, size))
同步(std::ios_base::sync_with_stdio)
语言环境处理(我们可以使用精简的语言环境，还是完全删除它?)

当然，欢迎使用其他方法.

注意:提到了 Dietmar Kuhl 的新"实现，但我无法找到有关它的许多细节.以前的参考文献似乎是死链接.

解决方案

这是我目前收集到的:

缓冲:

如果默认缓冲区很小，增加缓冲区大小肯定可以提高性能:

它减少了硬盘命中次数
减少系统调用次数

可以通过访问底层的streambuf实现来设置缓冲区.

char Buffer[N];std::ifstream 文件("file.txt");file.rdbuf()->pubsetbuf(Buffer, N);//保证 rdbuf 的指针读取器//构造函数成功后非空

@iavr 提供的警告:根据 cppreference最好在打开文件之前调用 pubsetbuf.否则，各种标准库实现具有不同的行为.

区域设置处理:

Locale 可以执行字符转换、过滤以及涉及数字或日期的更聪明的技巧.它们经历了一个复杂的动态调度和虚拟调用系统，因此删除它们有助于减少惩罚.

默认的 C 语言环境旨在不执行任何转换以及跨机器统一.这是一个很好的默认使用.

同步:

我看不到使用此工具的任何性能改进.

可以使用sync_with_stdio 静态函数访问全局设置(std::ios_base 的静态成员).

测量:

玩这个，我玩弄了一个简单的程序，使用 gcc 3.4.2 在 SUSE 10p3 和 -O2 上编译.

<块引用>

C : 7.76532e+06
C++:1.0874e+07

对于默认代码，这代表了大约 20%... 的减速.实际上，篡改缓冲区(在 C 或 C++ 中)或同步参数 (C++) 并没有产生任何改进.

其他人的结果:

<块引用>

@Irfy on g++ 4.7.2-2ubuntu1，-O3，虚拟化 Ubuntu 11.10，3.5.0-25-generic，x86_64，足够的 ram/cpu，196MB 的几个find/>> largefile.txt"运行

C:634572C++:473222

C++ 快 25%

<块引用>

@Matteo Italia 在 g++ 4.4.5、-O3、Ubuntu Linux 10.10 x86_64 上使用一个随机的 180 MB 文件

电话:910390
C++:776016

C++ 快 17%

<块引用>

@Bogatyr on g++ i686-apple-darwin10-g++-4.2.1 (GCC) 4.2.1(Apple Inc. build 5664)，mac mini，4GB 内存，空闲，除了这个测试有 168MB 数据文件

C : 4.34151e+06
C++:9.14476e+06

C++ 慢 111%

<块引用>

@Asu on clang++ 3.8.0-2ubuntu4、Kubuntu 16.04 Linux 4.8-rc3、8GB ram、i5 Haswell、Crucial SSD、88MB 数据文件(tar.xz 存档)

C : 270895C++:162799

C++ 快 66%

所以答案是:这是一个实施质量问题，实际上取决于平台:/

对于那些对基准测试感兴趣的人，这里有完整的代码:

#include #include #include #include <cmath>#include #include 模板 双基准(Func f，size_t 迭代){F();时间值 a, b;gettimeofday(&a, 0);for (; 迭代 --> 0;){F();}gettimeofday(&b, 0);返回 (b.tv_sec * (unsigned int)1e6 + b.tv_usec) -(a.tv_sec * (unsigned int)1e6 + a.tv_usec);}结构 CRead{CRead(char const* 文件名): _filename(filename) {}无效运算符()(){FILE* file = fopen(_filename, "r");整数计数 = 0;while ( fscanf(file,"%s", _buffer) == 1 ) { ++count;}fclose(文件)；}字符常量* _文件名；字符_缓冲区[1024]；};结构 CppRead{CppRead(char const* filename): _filename(filename), _buffer() {}枚举 { BufferSize = 16184 };无效运算符()(){std::ifstream 文件(_filename, std::ifstream::in);//删除扩展缓冲区的注释file.rdbuf()->pubsetbuf(_buffer, BufferSize);整数计数 = 0;std::string s;而(文件>> s){ ++计数;}}字符常量* _文件名；char _buffer[BufferSize];};int main(int argc, char* argv[]){size_t 迭代次数 = 1;if (argc > 1) { 迭代 = atoi(argv[1]);}char const* oldLocale = setlocale(LC_ALL,"C");如果(strcmp(oldLocale，C")！= 0){std::cout <<替换了旧的语言环境"<<oldLocale<<"' by 'C'
";}char const* 文件名 = "大文件.txt";CRead 读取(文件名)；CppRead cppread(文件名);//注释以使用默认设置bool oldSyncSetting = std::ios_base::sync_with_stdio(false);双 ctime = 基准(创建，迭代)；双 cpptime = 基准(cppread，迭代)；//如果注释了 oldSyncSetting 的声明，则注释std::ios_base::sync_with_stdio(oldSyncSetting);std::cout <<C:"<<时间<<"
"C++:"<<cpptime<<"
";返回0；}

Most C++ users that learned C prefer to use the printf / scanf family of functions even when they're coding in C++.

Although I admit that I find the interface way better (especially POSIX-like format and localization), it seems that an overwhelming concern is performance.

Taking at look at this question:

How can I speed up line by line reading of a file

It seems that the best answer is to use fscanf and that the C++ ifstream is consistently 2-3 times slower.

I thought it would be great if we could compile a repository of "tips" to improve IOStreams performance, what works, what does not.

Points to consider

buffering (rdbuf()->pubsetbuf(buffer, size))
synchronization (std::ios_base::sync_with_stdio)
locale handling (Could we use a trimmed-down locale, or remove it altogether ?)

Of course, other approaches are welcome.

Note: a "new" implementation, by Dietmar Kuhl, was mentioned, but I was unable to locate many details about it. Previous references seem to be dead links.

解决方案

Here is what I have gathered so far:

Buffering:

If by default the buffer is very small, increasing the buffer size can definitely improve the performance:

it reduces the number of HDD hits
it reduces the number of system calls

Buffer can be set by accessing the underlying streambuf implementation.

char Buffer[N];

std::ifstream file("file.txt");

file.rdbuf()->pubsetbuf(Buffer, N);
// the pointer reader by rdbuf is guaranteed
// to be non-null after successful constructor

Warning courtesy of @iavr: according to cppreference it is best to call pubsetbuf before opening the file. Various standard library implementations otherwise have different behaviors.

Locale Handling:

Locale can perform character conversion, filtering, and more clever tricks where numbers or dates are involved. They go through a complex system of dynamic dispatch and virtual calls, so removing them can help trimming down the penalty hit.

The default C locale is meant not to perform any conversion as well as being uniform across machines. It's a good default to use.

Synchronization:

I could not see any performance improvement using this facility.

One can access a global setting (static member of std::ios_base) using the sync_with_stdio static function.

Measurements:

Playing with this, I have toyed with a simple program, compiled using gcc 3.4.2 on SUSE 10p3 with -O2.

C : 7.76532e+06
C++: 1.0874e+07

Which represents a slowdown of about 20%... for the default code. Indeed tampering with the buffer (in either C or C++) or the synchronization parameters (C++) did not yield any improvement.

Results by others:

@Irfy on g++ 4.7.2-2ubuntu1, -O3, virtualized Ubuntu 11.10, 3.5.0-25-generic, x86_64, enough ram/cpu, 196MB of several "find / >> largefile.txt" runs

C : 634572 C++: 473222

C++ 25% faster

@Matteo Italia on g++ 4.4.5, -O3, Ubuntu Linux 10.10 x86_64 with a random 180 MB file

C : 910390
C++: 776016

C++ 17% faster

@Bogatyr on g++ i686-apple-darwin10-g++-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5664), mac mini, 4GB ram, idle except for this test with a 168MB datafile

C : 4.34151e+06
C++: 9.14476e+06

C++ 111% slower

@Asu on clang++ 3.8.0-2ubuntu4, Kubuntu 16.04 Linux 4.8-rc3, 8GB ram, i5 Haswell, Crucial SSD, 88MB datafile (tar.xz archive)

C : 270895 C++: 162799

C++ 66% faster

So the answer is: it's a quality of implementation issue, and really depends on the platform :/

The code in full here for those interested in benchmarking:

#include <fstream>
#include <iostream>
#include <iomanip>

#include <cmath>
#include <cstdio>

#include <sys/time.h>

template <typename Func>
double benchmark(Func f, size_t iterations)
{
  f();

  timeval a, b;
  gettimeofday(&a, 0);
  for (; iterations --> 0;)
  {
    f();
  }
  gettimeofday(&b, 0);
  return (b.tv_sec * (unsigned int)1e6 + b.tv_usec) -
         (a.tv_sec * (unsigned int)1e6 + a.tv_usec);
}


struct CRead
{
  CRead(char const* filename): _filename(filename) {}

  void operator()() {
    FILE* file = fopen(_filename, "r");

    int count = 0;
    while ( fscanf(file,"%s", _buffer) == 1 ) { ++count; }

    fclose(file);
  }

  char const* _filename;
  char _buffer[1024];
};

struct CppRead
{
  CppRead(char const* filename): _filename(filename), _buffer() {}

  enum { BufferSize = 16184 };

  void operator()() {
    std::ifstream file(_filename, std::ifstream::in);

    // comment to remove extended buffer
    file.rdbuf()->pubsetbuf(_buffer, BufferSize);

    int count = 0;
    std::string s;
    while ( file >> s ) { ++count; }
  }

  char const* _filename;
  char _buffer[BufferSize];
};


int main(int argc, char* argv[])
{
  size_t iterations = 1;
  if (argc > 1) { iterations = atoi(argv[1]); }

  char const* oldLocale = setlocale(LC_ALL,"C");
  if (strcmp(oldLocale, "C") != 0) {
    std::cout << "Replaced old locale '" << oldLocale << "' by 'C'
";
  }

  char const* filename = "largefile.txt";

  CRead cread(filename);
  CppRead cppread(filename);

  // comment to use the default setting
  bool oldSyncSetting = std::ios_base::sync_with_stdio(false);

  double ctime = benchmark(cread, iterations);
  double cpptime = benchmark(cppread, iterations);

  // comment if oldSyncSetting's declaration is commented
  std::ios_base::sync_with_stdio(oldSyncSetting);

  std::cout << "C  : " << ctime << "
"
               "C++: " << cpptime << "
";

  return 0;
}

相关文章