Python:读取 12 位二进制文​​件

2022-01-09 00:00:00 python python-3.x binary unpack

问题描述

我正在尝试使用 Python 3 读取包含图像(视频)的 12 位二进制文​​件.

I am trying to read 12-bit binary files containing images (a video) using Python 3.

要读取类似的文件，但以 16 位编码，以下方法非常有效:

To read a similar file but encoded in 16 bits, the following works very well:

import numpy as np
images = np.memmap(filename_video, dtype=np.uint16, mode='r', shape=(nb_frames, height, width))

其中 filename_video 是文件，nb_frames 是可以从另一个文件中读取的视频的高度和宽度特征.运行良好"是指快速:在我的计算机上读取 140 帧的 640x256 视频大约需要 1 毫秒.

where filename_video is the file and nb_frames, height, and width characteristics of the video that can be read from another file. By 'working very well' I mean fast: reading a 640x256 video that has 140 frames takes about 1 ms on my computer.

据我所知，当文件以 12 位编码时，我不能使用它，因为没有 uint12 类型.所以我要做的是读取一个 12 位文件并将其存储在一个 16 位 uint 数组中.以下内容取自 (Python:读取 12 位打包二进制图像)，作品:

As far as I know I cannot use this when the file is encoded in 12 bits because there is no uint12 type. So what I am trying to do is to read a 12-bit file and store it in a 16-bit uint array. The following, taken from (Python: reading 12 bit packed binary image), works:

with open(filename_video, 'rb') as f:
    data=f.read()
images=np.zeros(int(2*len(data)/3),dtype=np.uint16)
ii=0
for jj in range(0,int(len(data))-2,3):
    a=bitstring.Bits(bytes=data[jj:jj+3],length=24)
    images[ii],images[ii+1] = a.unpack('uint:12,uint:12')
    ii=ii+2
images = np.reshape(images,(nb_frames,height,width))

但是，这非常慢:使用我的机器读取只有 5 帧的 640x256 视频大约需要 11.5 秒.理想情况下，我希望能够像使用 memmap 读取 8 或 16 位文件一样有效地读取 12 位文件.或者至少不会慢 10^5 倍.我怎样才能加快速度?

However, this is very slow: reading a 640x256 video thas has only 5 frames takes about 11.5 s with my machine. Ideally I would like to be able to read 12-bit files as efficiently as I can read 8 or 16-bit files using memmap. Or at least not 10^5 times slower. How could I speed things up ?

这是一个文件示例:http://s000.tinyupload.com/index.php?file_id=26973488795334213426(nb_frames=5，高度=256，宽度=640).

Here is a file example: http://s000.tinyupload.com/index.php?file_id=26973488795334213426 (nb_frames=5, height=256, width=640).

解决方案

我的实现与@max9111 提出的实现略有不同，它不需要调用 unpackbits.

I have a slightly different implementation from the one proposed by @max9111 that doesn't require a call to unpackbits.

它通过将中间字节切成两半并使用 numpy 的二进制操作直接从三个连续的 uint8 创建两个 uint12 值.在下文中，data_chunks 假定为二进制字符串，其中包含任意数量的 12 位整数的信息(因此其长度必须是 3 的倍数).

It creates two uint12 values from three consecutive uint8 directly by cutting the middle byte in half and using numpy's binary operations. In the following, data_chunks is assumed to be a binary string containing the information for an arbitrary number number of 12-bit integers (hence its length must be a multiple of 3).

def read_uint12(data_chunk):
    data = np.frombuffer(data_chunk, dtype=np.uint8)
    fst_uint8, mid_uint8, lst_uint8 = np.reshape(data, (data.shape[0] // 3, 3)).astype(np.uint16).T
    fst_uint12 = (fst_uint8 << 4) + (mid_uint8 >> 4)
    snd_uint12 = ((mid_uint8 % 16) << 8) + lst_uint8
    return np.reshape(np.concatenate((fst_uint12[:, None], snd_uint12[:, None]), axis=1), 2 * fst_uint12.shape[0])

我对其他实现进行了基准测试，事实证明这种方法在大约 5 Mb 的输入上速度提高了大约 4 倍:
read_uint12_unpackbits 每个循环 65.5 ms ± 1.11 ms(平均值 ± 标准差，7 次运行，每次 10 个循环)read_uint12 每个循环 14 ms ± 513 µs(平均值 ± 标准偏差，7 次运行，每次 100 个循环)

I benchmarked with the other implementation and this approach proved to be ~4x faster on a ~5 Mb input:
read_uint12_unpackbits 65.5 ms ± 1.11 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) read_uint12 14 ms ± 513 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

相关文章