L2S-Training with Continuous Sensor System Parameters and Irregular Data

学习感知（L2S）研究单元的主要思想是与神经网络一起共同优化传感器系统的设计参数，以分析特定任务的结果数据。传感器系统设计中的自由度包括传感器的空间和光谱布局以及像素形状、传感器系统的光学、主动照明的选择以及片上处理能力。我们期望传感器系统和神经网络的联合、真正的端到端优化在最终性能、内存和功耗方面的效率以及对记录和注释训练数据的渴望方面具有显著的优势。这个子项目将研究实现上述L2S概念的几个基本机器学习方面。首先，一旦传感器布局本身进行了优化，分析生成数据的网络架构需要适应数据几何分布的变化。因此，我将研究如何使用图和函数值神经网络来处理优化过程中的表示变化。除了输入，神经网络输出的表示也需要解决。其次，每当传感器系统的数据形成过程非常复杂并且不能产生经典图像数据时，例如在编码光学和照明的情况下，我希望将物理知识纳入网络架构是至关重要的。向网络提供这些知识应该可以避免对已知物理关系的渴求数据的学习。第三，一旦选择了体系结构，网络和传感器系统参数的联合优化就是非平凡的。我将针对具有可微分渲染器的连续自适应传感器系统参数的情况解决此优化问题，并特别关注如何将真实数据包含到训练过程中。最后，本子项目将考虑通过双级优化问题分离训练网络和传感器系统参数的数据集，从而提高泛化的训练方案。上述基本机器学习问题将在RGB、太赫兹（THz）和微观成像传感器系统上的分类、定位和重建问题中得到举例说明。

The main idea of the Learning to Sense (L2S) research unit is to jointly optimize design parameters of a sensor system along with a neural network to analyze the resulting data on specific tasks. Degrees of freedom in the sensor system design include the spatial and spectral layout of the sensor as well as the pixel shape, the optics of the sensor system, choices of active illumination, and on-chip processing capabilities. We expect a joint, true end-to-end optimization of sensor systems and neural networks to have significant benefits in terms of the resulting performance, its efficiency with respect to memory and power consumption, and its hunger for recorded and annotated training data. This subproject will study several fundamental machine learning aspects for realizing the aforementioned L2S concept. First, as soon as the sensor layout itself is subject to an optimization, the network architectures analyzing the produced data need to be adaptive to changes in the geometric distribution of the data. Thus, I will study how graph and function-valued neural networks can be used to handle the representation changes during the optimization. Besides the input, the representation of the output of the neural network needs to be addressed. Second, whenever the data formation process of the sensor system is highly complex and does not yield classical image data, e.g. in the case of coded optics and illumination, I expect the inclusion of physical knowledge into the network architecture to be crucial. Providing the network with such knowledge should avoid a data-hungry learning of known physical relations.Third, once the architecture is chosen, the joint optimization for network and sensor system parameters is non-trivial. I will address this optimization for the case of continuously adaptable sensor system parameters with differentiable renderers and lay a particular focus on how real data can be included into the training process. Finally, this subproject will consider training schemes that aim at an improved generalization by separating the data sets the network and sensor system parameters are trained on via a bi-level optimization problem. The above fundamental machine learning questions will be exemplified in classification, localization, and reconstruction problems on RGB, Terahertz (THz), and microscopic imaging sensor systems.