Forward and Differentiable Simulation of L2S Sensor Data

学习感知（L2 S）研究单元的总体目标，即，传感器系统的设计参数和相关联的神经网络的联合优化以端到端机器学习的方式分析结果数据需要大量用于不同传感器和场景配置的训练数据。由于用真实的传感器收集训练和测试数据是昂贵的，或者部分根本不可能，所以传感器数据形成过程的仿真是在传感器系统参数和给定的应用任务之间建立联系的关键成功因素。在场景中以不同的波长（视觉在0.4-0.7µm之间，太赫兹（THz）在0.4- 0.6 mm之间）出现的真实世界效果，频率可能高达10³。这还包括使用复折射率和合成的，即，非聚焦成像方法和传感器系统设计的方面，例如，从技术上讲，该项目的重点是传感器数据的前向和可微模拟，从而实现基于机器学习的应用程序和硬件开发，以适应任意传感器和场景参数。在这方面，三个主要方面进行了调查。首先，被认为是一个模拟框架的设计和开发，能够模拟聚焦成像在视觉领域和非聚焦相干太赫兹辐射，包括入射辐射的传感器数据的转换。第二个研究重点涉及扩展现有的前向仿真方法，以实现高性能的路径跟踪仿真技术，产生物理上合理的传感器输出，并允许端到端的场景和传感器参数映射到所产生的光电特性。为了实现这些目标，该项目将与硬件项目P4（传感器布局和片上计算）、P6（波动光学效应模拟）和P7（相干太赫兹辐射和材料相互作用模拟）密切合作。此外，与机器学习项目P1在处理不规则传感器布置方面，P2在图像分类和语义分割方面，以及P3在THz频域中直接定位和对象重建方面进行了密切合作。

The overall goal of the Learning to Sense (L2S) research unit, i.e., the joint optimization of the design parameters of a sensor system and the associated neural network to analyze the resulting data in an end-to-end machine learning fashion, requires a large amount of training data for different sensor and scene configurations. Since collecting training and test data with real sensors is costly or partially not possible at all, simulation of the sensor data formation process is a key success factor to establish the link between sensor system parameters and the given application task.This subproject focuses on the efficient simulation of the sensor data formation process, which includes the simulation of physical, real-world effects occurring in the scene at different wavelengths (visual between 0.4-0.7µm and terahertz (THz) between 0.4-0.6mm) for a potentially large number of up to 10³ frequencies. This also includes the simulation of coherent radiation and material interaction using complex refractive indices and synthetic, i.e., unfocused imaging methods, and aspects of sensor system design, e.g., pixel and spectral filter placement.Technically, the focus of this project is on forward as well as differentiable simulation of sensor data, enabling application and hardware development based on machine learning for arbitrary sensor and scene parameters. In this context, three main aspects are investigated. First, the design and development of a simulation framework capable of simulating both focused imaging in the visual domain and unfocused coherent THz radiation, including the conversion of incident radiation to sensor data, is considered. The second research focus relates to the extension of existing forward simulation approaches to achieve high-performance path-tracing simulation techniques that produce physically plausible sensor outputs and allows end-to-end mapping of scene and sensor parameters to the resulting photoelectric properties. Third, efficient differentiable methods for the simulation process will be developed to support machine learning to efficiently identify optimal sensor parameters.To achieve these goals, the project will work closely with hardware projects P4 on sensor layout and on-chip calculations, P6 on simulation of wave-optical effects, and P7 regarding simulation of coherent THz- radiation and material interaction. In addition, there is intensive collaboration with machine learning projects P1 regarding the handling of irregular sensor arrangements, P2 on image classification and semantic segmentation, and P3 regarding localization and object reconstruction directly in the THz frequency domain.