Advanced Image Sensing Using Arbitrarily Shaped Pixels and Neural Network Reconstruction

本项目的目标是系统地提出具有不规则形状像素的新型图像传感器的概念，并设计基于神经网络的图像重建技术。因此，与常规正方形像素相比，结合最先进的单幅图像超分辨率算法，在保持相同样本数量的情况下，图像质量将得到显着改善。非规则采样比规则采样具有优势，因为它可以产生更高的每采样像素分辨率。图像处理中的一个应用是称为1/4采样的传感器概念。在1/4采样中，图像传感器的每个正方形像素都被覆盖，以便只有一个随机选择的象限保持透明。由于填充因子以这种方式减少了四倍，并且在这样的实现中损失了75%的入射光，因此寻找了解决方案以在增加填充因子的同时保持非规则性。本项目研究的一种有希望的方法是使用非规则形状的像素以100%的填充系数覆盖整个传感器区域。对于这项任务，有几个可用的领域是有希望的。与正方形像素的常规传感器不同，测量数据需要经过处理才能在规则网格上重建图像。虽然已知针对类似任务的基于模型的方法，但我们计划将其扩展到神经网络的使用，因为这些方法在相关研究领域显示了令人振奋的结果。神经网络用于非规则抽样的研究一直很少，但前景看好。因此，新的传感器布局与新的基于神经网络的重建方法相结合，具有开发新型更高分辨率相机传感器的巨大潜力。

Objective of this project is the systematic conception of novel image sensors with non-regularly shaped pixels and the design of neural network-based image reconstruction techniques. Thereby the image quality shall be significantly improved compared to regular square pixels in combination with state-of-the-art single-image super-resolution algorithms while retaining the same number of samples. Non-regular sampling has advantages over regular sampling as it can result in higher resolution per sampled pixel. An application in image processing is a sensor concept called 1/4 sampling. In 1/4 sampling, each square pixel of an image sensor is covered such that only one randomly chosen quadrant is left transparent. Since the fill factor is decreased by a factor of four that way, and 75 % of the incoming light is lost in such an implementation, solutions were searched to keep the non-regularity while increasing the fill factor. One promising way that is investigated in this project is the usage of non-regularly shaped pixels covering the full sensor area at 100 % fill factor. There are several tilings of areas available that are promising for this task. In contrast to a regular sensor with square pixels, the measured data needs to be processed to reconstruct the image on a regular grid. While a model-based approach for similar tasks is known, we are planning to expand this to the usage of neural networks as these have shown promising results in related fields of research. The use of neural networks for non-regular sampling has been poorly investigated and is promising. Such, the new sensor layouts combined with new neural network-based reconstruction methods have great potential for novel higher resolution camera sensors.