L2S-Training with Continuous Sensor System Parameters and Irregular Data

学习感知(L2S)研究单元的主要思想是联合优化传感器系统的设计参数和神经网络，以分析特定任务的结果数据。传感器系统设计中的自由度包括传感器的空间和光谱布局以及像素形状、传感器系统的光学特性、主动照明的选择以及片上处理能力。我们预计，传感器系统和神经网络的联合、真正的端到端优化将在所产生的性能、其在内存和功率消耗方面的效率以及对记录和注释的训练数据的需求方面具有显著的好处。这个子项目将研究实现上述L2S概念的几个基本机器学习方面。首先，一旦传感器布局本身受到优化，分析产生的数据的网络体系结构需要适应数据几何分布的变化。因此，我将研究如何使用图形和函数值神经网络来处理优化过程中的表示变化。除了输入之外，还需要解决神经网络输出的表示。其次，每当传感器系统的数据形成过程非常复杂并且不产生经典图像数据时，例如在编码光学和照明的情况下，我预计将物理知识包括到网络体系结构中将是至关重要的。为网络提供这样的知识应该避免对已知物理关系的数据饥渴学习。第三，一旦选择了体系结构，网络和传感器系统参数的联合优化就不是微不足道的了。我将针对具有不同呈现器的连续自适应传感器系统参数的情况进行这种优化，并特别关注如何将真实数据包括到训练过程中。最后，这个子项目将通过一个双层优化问题来考虑旨在通过分离网络和传感器系统参数训练的数据集来改进泛化的训练方案。上述基本的机器学习问题将在RGB、太赫兹(THz)和显微成像传感器系统的分类、定位和重建问题中得到例证。

The main idea of the Learning to Sense (L2S) research unit is to jointly optimize design parameters of a sensor system along with a neural network to analyze the resulting data on specific tasks. Degrees of freedom in the sensor system design include the spatial and spectral layout of the sensor as well as the pixel shape, the optics of the sensor system, choices of active illumination, and on-chip processing capabilities. We expect a joint, true end-to-end optimization of sensor systems and neural networks to have significant benefits in terms of the resulting performance, its efficiency with respect to memory and power consumption, and its hunger for recorded and annotated training data. This subproject will study several fundamental machine learning aspects for realizing the aforementioned L2S concept. First, as soon as the sensor layout itself is subject to an optimization, the network architectures analyzing the produced data need to be adaptive to changes in the geometric distribution of the data. Thus, I will study how graph and function-valued neural networks can be used to handle the representation changes during the optimization. Besides the input, the representation of the output of the neural network needs to be addressed. Second, whenever the data formation process of the sensor system is highly complex and does not yield classical image data, e.g. in the case of coded optics and illumination, I expect the inclusion of physical knowledge into the network architecture to be crucial. Providing the network with such knowledge should avoid a data-hungry learning of known physical relations.Third, once the architecture is chosen, the joint optimization for network and sensor system parameters is non-trivial. I will address this optimization for the case of continuously adaptable sensor system parameters with differentiable renderers and lay a particular focus on how real data can be included into the training process. Finally, this subproject will consider training schemes that aim at an improved generalization by separating the data sets the network and sensor system parameters are trained on via a bi-level optimization problem. The above fundamental machine learning questions will be exemplified in classification, localization, and reconstruction problems on RGB, Terahertz (THz), and microscopic imaging sensor systems.