Forward and Differentiable Simulation of L2S Sensor Data

学习感知(L2S)研究单元的总体目标，即联合优化传感器系统和相关神经网络的设计参数，以端到端的机器学习方式分析结果数据，需要针对不同传感器和场景配置的大量训练数据。由于使用真实传感器收集训练和测试数据成本高昂或部分根本不可能，因此传感器数据形成过程的模拟是在传感器系统参数和给定应用任务之间建立联系的关键成功因素。该子项目专注于传感器数据形成过程的有效模拟，其中包括对场景中不同波长(视觉在0.4-0.7微米和太赫兹(THz)在0.4-0.6毫米之间)下发生的物理真实效果的模拟。该项目还包括使用复折射率和合成方法(即非聚焦成像方法)模拟相干辐射和材料相互作用，以及传感器系统设计的各个方面，例如像素和光谱滤光片放置。在技术上，该项目的重点是传感器数据的正向和可区分模拟，使基于机器学习的应用程序和硬件开发能够针对任意传感器和场景参数。在此背景下，主要从三个方面进行了研究。首先，设计和开发了一种仿真框架，能够同时模拟视觉域聚焦成像和非聚焦相干太赫兹辐射，包括入射辐射到传感器数据的转换。第二个研究重点是扩展现有的正向模拟方法，以实现高性能的路径跟踪模拟技术，这些模拟技术产生物理上可信的传感器输出，并允许将场景和传感器参数端到端映射到所产生的光电特性。为了实现这些目标，该项目将与硬件项目P4在传感器布局和片上计算方面密切合作，在波光效应模拟方面与P6密切合作，在相干太赫兹辐射和材料相互作用模拟方面与P7密切合作。此外，在处理不规则传感器布置方面与机器学习项目P1进行了密集协作，在图像分类和语义分割方面与P2进行了密切合作，在太赫兹频域直接与机器学习项目P3进行了定位和目标重建。

The overall goal of the Learning to Sense (L2S) research unit, i.e., the joint optimization of the design parameters of a sensor system and the associated neural network to analyze the resulting data in an end-to-end machine learning fashion, requires a large amount of training data for different sensor and scene configurations. Since collecting training and test data with real sensors is costly or partially not possible at all, simulation of the sensor data formation process is a key success factor to establish the link between sensor system parameters and the given application task.This subproject focuses on the efficient simulation of the sensor data formation process, which includes the simulation of physical, real-world effects occurring in the scene at different wavelengths (visual between 0.4-0.7µm and terahertz (THz) between 0.4-0.6mm) for a potentially large number of up to 10³ frequencies. This also includes the simulation of coherent radiation and material interaction using complex refractive indices and synthetic, i.e., unfocused imaging methods, and aspects of sensor system design, e.g., pixel and spectral filter placement.Technically, the focus of this project is on forward as well as differentiable simulation of sensor data, enabling application and hardware development based on machine learning for arbitrary sensor and scene parameters. In this context, three main aspects are investigated. First, the design and development of a simulation framework capable of simulating both focused imaging in the visual domain and unfocused coherent THz radiation, including the conversion of incident radiation to sensor data, is considered. The second research focus relates to the extension of existing forward simulation approaches to achieve high-performance path-tracing simulation techniques that produce physically plausible sensor outputs and allows end-to-end mapping of scene and sensor parameters to the resulting photoelectric properties. Third, efficient differentiable methods for the simulation process will be developed to support machine learning to efficiently identify optimal sensor parameters.To achieve these goals, the project will work closely with hardware projects P4 on sensor layout and on-chip calculations, P6 on simulation of wave-optical effects, and P7 regarding simulation of coherent THz- radiation and material interaction. In addition, there is intensive collaboration with machine learning projects P1 regarding the handling of irregular sensor arrangements, P2 on image classification and semantic segmentation, and P3 regarding localization and object reconstruction directly in the THz frequency domain.