使用 tensorflow 数据集的 GPU 利用率低下

2022-01-21 00:00:00 python tensorflow dataset

问题描述

在我的数据训练期间，我的 GPU 利用率约为 40%，我清楚地看到有一个基于 tensorflow 分析器的数据复制操作占用了大量时间(见附图).我认为MEMCPYHtoD"选项正在将批处理从 CPU 复制到 GPU，并阻止使用 GPU.无论如何将数据预取到GPU?还是有其他我没有看到的问题?

During training of my data, my GPU utilization is around 40%, and I clearly see that there is a datacopy operation that's using a lot of time, based on tensorflow profiler(see attached picture). I presume that "MEMCPYHtoD" option is copying the batch from CPU to GPU, and is blocking the GPU from being used. Is there anyway to prefetch data to GPU? or is there other problems that I am not seeing?

这里是数据集的代码:

X_placeholder = tf.placeholder(tf.float32, data.train.X.shape)
y_placeholder = tf.placeholder(tf.float32, data.train.y[label].shape)

dataset = tf.data.Dataset.from_tensor_slices({"X": X_placeholder, 
                                              "y": y_placeholder})
dataset = dataset.repeat(1000)
dataset = dataset.batch(1000)
dataset = dataset.prefetch(2)
iterator = dataset.make_initializable_iterator()
next_element = iterator.get_next()

解决方案

预取到单个 GPU:

  • 考虑使用比 prefetch_to_device 更灵活的方法，例如通过使用 tf.data.experimental.copy_to_device(...) 显式复制到 GPU，然后进行预取.这允许避免 prefetch_to_device 必须是管道中的最后一个转换的限制，并允许结合进一步的技巧来优化 Dataset 管道性能(例如 通过覆盖线程池分布).
  • 试用实验性的 tf.contrib.data.AUTOTUNE 选项进行预取，它允许 tf.data 运行时根据您的系统自动调整预取缓冲区大小和环境.
  • Consider using a more flexible approach than prefetch_to_device, e.g. by explicitly copying to the GPU with tf.data.experimental.copy_to_device(...) and then prefetching. This allows to avoid the restriction that prefetch_to_device must be the last transformation in a pipeline, and allow to incorporate further tricks to optimize the Dataset pipeline performance (e.g. by overriding threadpool distribution).
  • Try out the experimental tf.contrib.data.AUTOTUNE option for prefetching, which allows the tf.data runtime to automatically tune the prefetch buffer sizes based on your system and environment.

最后，你可能会做这样的事情:

At the end, you might end up doing something like this:

dataset = dataset.apply(tf.data.experimental.copy_to_device("/gpu:0"))
dataset = dataset.prefetch(tf.contrib.data.AUTOTUNE)

相关文章