Advanced Image Sensing Using Arbitrarily Shaped Pixels and Neural Network Reconstruction

本计画的目的是系统性的构想具有不规则形状像素的新颖影像感测器，以及设计以神经网路为基础的影像重建技术。因此，与常规正方形像素相比，结合最先进的单图像超分辨率算法，图像质量将得到显著改善，同时保留相同数量的样本。非规则采样具有优于规则采样的优点，因为它可以导致每个采样像素的更高分辨率。图像处理中的一个应用是称为1/4采样的传感器概念。在1/4采样中，图像传感器的每个正方形像素被覆盖，使得仅一个随机选择的象限保持透明。由于填充因子以这种方式减小四倍，并且在这种实施方式中损失了75%的入射光，因此寻找解决方案以在增加填充因子的同时保持非规则性。在该项目中研究的一种有前途的方法是使用非规则形状的像素，以100%的填充因子覆盖整个传感器区域。有几个可用的区域的平铺是有希望的这项任务。与具有正方形像素的常规传感器相比，需要处理测量数据以在常规网格上重建图像。虽然类似任务的基于模型的方法是已知的，但我们计划将其扩展到神经网络的使用，因为这些在相关研究领域已经显示出有希望的结果。神经网络用于非规则采样的研究很少，但很有前途。因此，新的传感器布局与新的基于神经网络的重建方法相结合，对于新型更高分辨率的相机传感器具有巨大的潜力。

Objective of this project is the systematic conception of novel image sensors with non-regularly shaped pixels and the design of neural network-based image reconstruction techniques. Thereby the image quality shall be significantly improved compared to regular square pixels in combination with state-of-the-art single-image super-resolution algorithms while retaining the same number of samples. Non-regular sampling has advantages over regular sampling as it can result in higher resolution per sampled pixel. An application in image processing is a sensor concept called 1/4 sampling. In 1/4 sampling, each square pixel of an image sensor is covered such that only one randomly chosen quadrant is left transparent. Since the fill factor is decreased by a factor of four that way, and 75 % of the incoming light is lost in such an implementation, solutions were searched to keep the non-regularity while increasing the fill factor. One promising way that is investigated in this project is the usage of non-regularly shaped pixels covering the full sensor area at 100 % fill factor. There are several tilings of areas available that are promising for this task. In contrast to a regular sensor with square pixels, the measured data needs to be processed to reconstruct the image on a regular grid. While a model-based approach for similar tasks is known, we are planning to expand this to the usage of neural networks as these have shown promising results in related fields of research. The use of neural networks for non-regular sampling has been poorly investigated and is promising. Such, the new sensor layouts combined with new neural network-based reconstruction methods have great potential for novel higher resolution camera sensors.