OpenCV Android - 使用 CameraBridgeViewBase 的颜色问题

2022-01-15 00:00:00 opencv macos android android-emulator java

我在使用 Android 模拟器时遇到了一个奇怪的问题.OpenCV CameraBridgeViewBase.

I'm encountering a strange problem using Android emulators & OpenCV CameraBridgeViewBase.

使用 onCameraFrame 我得到一张看起来没有正确解码的图片.

Using onCameraFrame I get a picture that looks like it wasn't decoded properly.

public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
    return inputFrame.rgba();
}

使用 'inputFrame.gray()' 我得到了预期的结果 - 没有伪影或任何其他问题的黑白图像.

Using 'inputFrame.gray()' I get what's expected - black and white image without artifacts or any other issues.

这就是我得到的:

再来一张(更大的)

到目前为止我已经尝试过:

What I've tried so far:

  1. 不同的 API 级别(从 15 到 21).
  2. 不同的模拟器:Genymotion 和谷歌安卓模拟器.
  3. 不同的平台架构 - ARM 和 Intel x86.
  4. 在装有 Linux 的其他笔记本电脑上启动模拟器:它可以正常工作，正如预期的那样，问题消失了！
  5. 使用从 Play 商店下载的 OpenCV 启动应用.他们确实有效！但是:
  1. Different API levels (from 15 up to 21).
  2. Different emulators: Genymotion & Google Android emulator.
  3. Different platform architectures - both ARM and Intel x86.
  4. Launching emulator on different laptop with Linux: it works as expected, the issue is gone!
  5. Launching apps, using OpenCV, downloaded from Play Store. They DO work! However:
  1. 启动按预期运行的应用，然后关闭它.
  2. 启动您的应用(或 OpenCV 教程之一)，然后关闭它.
  3. 再次从 5.1 启动应用程序，我发现它受到相同错误的影响！

  • 不同的 OpenCV 版本(2.4.9 和 2.4.10).
  • OpenCV 管理器的不同版本(一个来自 Play 商店，2.4.9 和 2.4.10 来自 OpenCV 包).
  • 最后，正如我在 5.2 中注意到的，OpenCV 包中的预编译教程 .apk 文件也受到该问题的影响.

    • 在我真正的 Android 设备上一切正常.

    Everything works as expected on my real android devices.

    查看 CameraBridgeViewBase 和 Java/Native 相机类的来源后，我决定在解码图像时出现问题.特定于平台的相机输出格式(YUV、NV21)可能存在问题.但是，奇怪的是 .gray() 给出了正确的图像(没有伪影).

    After looking at sources of the CameraBridgeViewBase and Java/Native camera classes I came to the decision that the problem occurs while decoding image. Probably there is a problem with platform-specific camera output format (YUV, NV21). However, it's strange that .gray() gives out a proper image (without artifacts).

    如果重要的话，我正在使用带有Facetime HD"摄像头的 Mac OS X 10.10 Yosemite 和 MacBook Air.

    I'm using Mac OS X 10.10 Yosemite and MacBook Air with "Facetime HD" camera if that matters.

    关于如何克服这个问题的任何想法&非常感谢您帮助找到问题的根源！

    Any ideas on how to overcome this problem & help in finding the root of the problem are greatly appreciated!

    推荐答案

    所以，在深入研究问题之后，我找到了问题的根源.

    So, after drilling into the problem I've found the root of the issue.

    让我们看一下OpenCV JavaCameraView 类及其CameraBridgeViewBase 基类.问题是在 onPreviewFrame 方法中作为 byte[] 数组接收的相机帧被错误解码.

    Let's take a look at the OpenCV JavaCameraView class and its CameraBridgeViewBase base class. The problem was that camera frames received as byte[] array in onPreviewFrame method were decoded incorrectly.

    发生解码过程的代码的确切位置是 JavaCameraView 的内部 JavaCameraFrame 类中的 Mat rgba() 方法的实现代码>:

    The exact place of code where the decoding process takes place is an implementation of the Mat rgba() method in inner JavaCameraFrame class of the JavaCameraView:

        public Mat rgba() {
        Imgproc.cvtColor(mYuvFrameData, mRgba, Imgproc.COLOR_YUV2RGBA_NV21, 4);
        return mRgba;
    }

    正如我们所见，Imgproc.cvtColor(...) 方法用于将帧从 YUV 转换为 RGBA.<代码>NV21 YUV ->RGBA 转换发生在那里.在初始化过程中，我们将格式设置为 NV21，所以这应该是正确的.此外，每台 Android 设备都应该支持 NV21.此外，我们可以使用调试器检查设备是否接受格式:

    As we see, Imgproc.cvtColor(...) method is used in order to convert frame from YUV to RGBA. NV21 YUV -> RGBA conversion takes place there. During the initialization process we set the format to NV21, so this should be right. Moreover, every Android device should support NV21. Also, we can check whether device accepted the format using debugger:

    protected boolean initializeCamera(int width, int height) {
    ...
    params.setPreviewFormat(ImageFormat.NV21);
    ...
    mCamera.setParameters(params);
    ...
    params = mCamera.getParameters();
    Log.d(TAG, String.format("Actual preview format is 0x%X", params.getPreviewFormat()));
}

    据报道，手机(HTC Sensation)和模拟器都在使用 NV21.

    Both phone (HTC Sensation) and emulator reported to be using NV21 indeed.

    但是，如果我们将 COLOR_YUV2RGBA_NV21 更改为 COLOR_YUV2RGB_I420(YV12 和 I420 是一样的，只是将 Y 和 V 反转；)我们会看到模拟器会得到终于有了合适的色彩空间.在 params.setPreviewFormat(ImageFormat.NV21); 中将 NV21 更改为 YV12 我们会得到类似的结果.看起来 Imgproc.cvtColor 或 Android 中存在错误.

    However, if we change COLOR_YUV2RGBA_NV21 to COLOR_YUV2RGB_I420 (YV12 and I420 is the same thing, just with Y and V inverted;) we'll see that emulator will get a proper color space finally. Changing NV21 to YV12 in params.setPreviewFormat(ImageFormat.NV21); we'll get similar results. Looks like there's bug either in Imgproc.cvtColor, or in Android.

    解决方案来了.修改public Mat rgba()如下:

        public Mat rgba() {
        if (previewFormat == ImageFormat.NV21) {
            Imgproc.cvtColor(mYuvFrameData, mRgba, Imgproc.COLOR_YUV2RGBA_NV21, 4);
        }
        else if (previewFormat == ImageFormat.YV12) {
            Imgproc.cvtColor(mYuvFrameData, mRgba, Imgproc.COLOR_YUV2RGB_I420, 4);  // COLOR_YUV2RGBA_YV12 produces inverted colors
        }
        return mRgba;
    }

    previewFormat 是一个新的 int 变量，它是这样声明的:

    previewFormat is a new int variable, it's declared this way:

    private int previewFormat = ImageFormat.NV21;

    在初始化中添加以下更改:

    Add following changes to the initialization:

    protected boolean initializeCamera(int width, int height) {
        ...
                params.setPreviewFormat(ImageFormat.NV21);
                // "generic" = android emulator
                if (Build.BRAND.equalsIgnoreCase("generic")) {
                    params.setPreviewFormat(ImageFormat.YV12);
                }
                ...
                mCamera.setParameters(params);
                params = mCamera.getParameters();
                previewFormat = params.getPreviewFormat();
        ...
}

    重要提示:
    请注意:这只是一个临时解决方案，以使 OpenCV 在我的情况下可与模拟器一起使用.应该做进一步的研究.在 onPreviewFrame 中检查设备是否使用正确的图像格式非常容易.有时间我会回到这个.

    Important:
    Please note: this is just a temporary solution in order to make OpenCV usable with emulator in my case. Further research should be done. It's quite easy to check whether device uses correct image format in onPreviewFrame. I'll get back to this when I have some time.

    相关文章