DXGI 桌面复制:编码帧以通过网络发送它们

2021-12-26 00:00:00 ffmpeg video-streaming c++ ms-media-foundation dxgi

我正在尝试编写一个应用程序，它将捕获屏幕的视频流并将其发送到远程客户端.我发现在 Windows 上捕获屏幕的最佳方法是使用 DXGI 桌面复制 API(自 Windows 8 起可用).Microsoft 提供了一个简洁的示例，可将重复的帧流式传输到屏幕.现在，我一直在想什么是对这些帧进行编码并通过网络发送的最简单但仍然相对较快的方法.

I'm trying to write an app which will capture a video stream of the screen and send it to a remote client. I've found out that the best way to capture a screen on Windows is to use DXGI Desktop Duplication API (available since Windows 8). Microsoft provides a neat sample which streams duplicated frames to screen. Now, I've been wondering what is the easiest, but still relatively fast way to encode those frames and send them over the network.

框架来自 AcquireNextFrame 带有一个包含桌面位图和元数据的表面，元数据包含更新的脏区和移动区域.从这里，我有几个选择:

The frames come from AcquireNextFrame with a surface that contains the desktop bitmap and metadata which contains dirty and move regions that were updated. From here, I have a couple of options:

  1. 从 DirectX 表面提取位图，然后使用像 ffmpeg 这样的外部库将一系列位图编码为 H.264 并通过 RTSP 发送.虽然很简单，但我担心这种方法会太慢，因为它没有利用任何本机 Windows 方法.将 D3D 纹理转换为与 ffmpeg 兼容的位图似乎是不必要的工作.
  2. 来自这个答案:将 D3D 纹理转换为 IMFSample 并使用 MediaFoundation 的 SinkWriter 对帧进行编码.我找到了本教程视频编码，但我还没有找到一种方法来立即获取编码帧并将其发送，而不是将它们全部转储到视频文件中.
  1. Extract a bitmap from a DirectX surface and then use an external library like ffmpeg to encode series of bitmaps to H.264 and send it over RTSP. While straightforward, I fear that this method will be too slow as it isn't taking advantage of any native Windows methods. Converting D3D texture to a ffmpeg-compatible bitmap seems like unnecessary work.
  2. From this answer: convert D3D texture to IMFSample and use MediaFoundation's SinkWriter to encode the frame. I found this tutorial of video encoding, but I haven't yet found a way to immediately get the encoded frame and send it instead of dumping all of them to a video file.

因为我以前没有做过这样的事情，所以我在问我是否朝着正确的方向前进.最后，我想要一个简单的，最好是低延迟的桌面捕获视频流，我可以从远程设备查看.

Since I haven't done anything like this before, I'm asking if I'm moving in the right direction. In the end, I want to have a simple, preferably low latency desktop capture video stream, which I can view from a remote device.

另外，我想知道我是否可以利用桌面复制提供的脏区域和移动区域.我可以不编码帧，而是通过网络发送它们并在客户端进行处理，但这意味着我的客户端必须有 DirectX 11.1 或更高版本可用，如果我想流式传输到移动平台，这是不可能的.

Also, I'm wondering if I can make use of dirty and move regions provided by Desktop Duplication. Instead of encoding the frame, I can send them over the network and do the processing on the client side, but this means that my client has to have DirectX 11.1 or higher available, which is impossible if I would want to stream to a mobile platform.

推荐答案

您可以使用 IMFTransform 接口进行 H264 编码.从 ID3D11Texture2D 获取 IMFSample 后，只需将其传递给 IMFTransform::ProcessInput 并从 IMFTransform::ProcessOutput 获取编码的 IMFSample.

You can use IMFTransform interface for H264 encoding. Once you get IMFSample from ID3D11Texture2D just pass it to IMFTransform::ProcessInput and get the encoded IMFSample from IMFTransform::ProcessOutput.

参考这个 编码示例详情.

获得编码的 IMFSamples 后，您可以通过网络一个一个地发送它们.

Once you get the encoded IMFSamples you can send them one by one over the network.

相关文章