Object Detection Based on Raw Bayer Images
Object Detection Based on Raw Bayer Images
复制标题
DOI：
10.1109/iros55552.2023.10342008
复制
发表时间：
2023-10
期刊：
影响因子：
--
通讯作者：
Guoyu Lu
中科院分区：
文献类型：
--
作者：
Guoyu Lu
拜耳图案是用于数字图像传感器的广泛使用的彩色滤清器阵列（CFA），可有效捕获不同像素上不同的光波长，而无需昂贵的ISP管道。由此产生的单渠道RAW BAYER图像提供了诸如光谱波长灵敏度和低时间延迟之类的优势。但是，由于相邻像素中不连续的颜色通道引起的人类观察和算法设计的挑战，基于拜耳图像的对象检测尚未逐渐变化。为了解决此问题，我们提出了BayerDect网络，这是一个端到端的深度对象检测框架，旨在实现快速，准确和内存有效的对象检测。与RGB颜色图像不同，每个像素在ISP颜色插值过程中编码来自相邻像素的光谱上下文，Raw Bayer图像缺乏光谱上下文。为了增强光谱环境，BayerDect网络引入了光谱频率注意力块，将原始的Bayer图像模式转换为频域。在对象检测中，清除对象边界对于准确的边界框预测至关重要。为了应对交替的光谱渠道所带来的挑战并减轻不连续边界的影响，BayerDect网络结合了一种空间注意方案，该方案在多个尺度上利用可变形的卷积内核来有效地探索空间上下文。然后，提取的卷积特征将通过一组稀疏的建议框进行检测和分类。我们在公共和自我收集的原始拜耳图像上进行了实验，结果证明了拜耳访问网络在对象检测任务中的出色性能。
Bayer pattern is a widely used Color Filter Array (CFA) for digital image sensors, efficiently capturing different light wavelengths on different pixels without the need for a costly ISP pipeline. The resulting single-channel raw Bayer images offer benefits such as spectral wavelength sensitivity and low time latency. However, object detection based on Bayer images has been underexplored due to challenges in human observation and algorithm design caused by the discontinuous color channels in adjacent pixels. To address this issue, we propose the BayerDetect network, an end-to-end deep object detection framework that aims to achieve fast, accurate, and memory-efficient object detection. Unlike RGB color images, where each pixel encodes spectral context from adjacent pixels during ISP color interpolation, raw Bayer images lack spectral context. To enhance the spectral context, the BayerDetect network introduces a spectral frequency attention block, transforming the raw Bayer image pattern to the frequency domain. In object detection, clear object boundaries are essential for accurate bounding box predictions. To handle the challenges posed by alternating spectral channels and mitigate the influence of discontinuous boundaries, the BayerDetect network incorporates a spatial attention scheme that utilizes deformable convolutional kernels in multiple scales to explore spatial context effectively. The extracted convolutional features are then passed through a sparse set of proposal boxes for detection and classification. We conducted experiments on both public and self-collected raw Bayer images, and the results demonstrate the superb performance of the BayerDetect network in object detection tasks.