Joint Demosaicing and Super-Resolution (JDSR): Network Design and Perceptual Optimization
Joint Demosaicing and Super-Resolution (JDSR): Network Design and Perceptual Optimization
复制标题
DOI：
10.1109/tci.2020.2999819
复制
发表时间：
2019-11
影响因子：
5.4
通讯作者：
Xuan Xu;Yanfang Ye;Xin Li
中科院分区：
文献类型：
--
作者：
Xuan Xu;Yanfang Ye;Xin Li
图像去马赛克和超分辨率是彩色图像处理中的两个重要任务。到目前为止，它们大多是在深度学习的开放文献中独立研究的;对于制定联合去马赛克和超分辨率（JDSR）问题的潜在好处知之甚少。在这篇文章中，我们提出了一个端到端的JDSR问题的优化解决方案，并证明其在计算成像的实际意义。我们的技术贡献主要有两方面。在网络设计上，我们开发了一个由预去马赛克网络（PDNet）作为预处理步骤支持的残差密集压缩和激励网络（RDSEN）。我们解决了彩色滤光片阵列（CFA）数据的空间光谱注意力的问题，并讨论了如何实现更好的信息流，通过串联剩余密集的挤压和激发块（RDSEBs）的JDSR。实验结果表明，显着的PSNR/SSIM增益可以实现RDSEN在以前的网络架构，包括国家的最先进的RCAN。在感知优化方面，我们提出利用最新的思想，包括相对论性的优化和预激励感知损失函数，以进一步提高重建图像中纹理区域的视觉质量。我们广泛的实验结果表明，纹理增强的相对论平均生成对抗网络（TRaGAN）可以产生主观上更令人愉快的图像，客观上比JDSR的标准GAN更低的感知失真分数。最后，我们已经验证了JDSR的好处，以高质量的图像重建从现实世界中的拜耳模式数据收集的NASA火星探测器。
Image demosaicing and super-resolution are two important tasks in color imaging pipeline. So far they have been mostly independently studied in the open literature of deep learning; little is known about the potential benefit of formulating a joint demosaicing and super-resolution (JDSR) problem. In this article, we propose an end-to-end optimization solution to the JDSR problem and demonstrate its practical significance in computational imaging. Our technical contributions are mainly two-fold. On network design, we have developed a Residual-Dense Squeeze-and-Excitation Networks (RDSEN) supported by a pre-demosaicing network (PDNet) as the pre-processing step. We address the issue of spatio-spectral attention for color-filter-array (CFA) data and discuss how to achieve better information flow by concatenating Residue-Dense Squeeze-and-Excitation Blocks (RDSEBs) for JDSR. Experimental results have shown that significant PSNR/SSIM gain can be achieved by RDSEN over previous network architectures including state-of-the-art RCAN. On perceptual optimization, we propose to leverage the latest ideas including relativistic discriminator and pre-excitation perceptual loss function to further improve the visual quality of textured regions in reconstructed images. Our extensive experiment results have shown that Texture-enhanced Relativistic average Generative Adversarial Network (TRaGAN) can produce both subjectively more pleasant images and objectively lower perceptual distortion scores than standard GAN for JDSR. Finally, we have verified the benefit of JDSR to high-quality image reconstruction from real-world Bayer pattern data collected by NASA Mars Curiosity.