Architectures for Non-Image-Generating L2S Vision Systems

学习感知（L2S）研究单元的主要目标是与神经网络一起优化特定应用传感器的设计参数，以分析结果数据。在这个子项目中，我们探索和实现L2S架构和机器学习（ML）算法，用于我们与研究小组的合作伙伴共同开发的许多新型传感器类型。这些传感器用于可见光、毫米波和太赫兹合成孔径雷达。这些传感器的共同特点是：(a)与相机和许多现有传感器不同，它们不是为生成图像而设计的，(b)它们使用差分、对比度测量。对于可见光，新的传感器部分是由人类视觉系统的发现激发的，它们在高动态范围环境中具有优于标准相机的潜力。我们建议建立具有中心环绕接受野和对手颜色编码的传感器（类似于视网膜神经节细胞），并计算芯片上模拟信号的空间和时间导数。在研究单元中，我们还开发和探索了注视点传感器，该子项目的重点是来自视野外围的差分低带宽数据。对于毫米波和太赫兹辐射，所提出的差分传感器消除了由于直接反射和多径散射而产生的大量背景信号，使它们有可能简化涉及计算机视觉的逆问题。虽然许多标准相机和传感器可以通过改进图像质量来优化，但非图像生成传感器只能在具有特定ML任务的端到端应用场景中进行优化和训练。我们在人脸和行人检测、光流、周边视觉以及毫米波和太赫兹SAR中结构的定位等背景下探索了这些设置。本项目的结果有助于L2S范式的总体发展，并为计算机视觉提供了几种新的实验方法。

The main goal of the Learning-to-Sense (L2S) research unit is to jointly optimize the design parameters of an application-specific sensor along with a neural network to analyze the resulting data.In this subproject, we explore and implement L2S architectures and machine learning (ML) algorithms for a number of novel sensor types that we develop together with partners in the research group. These sensors are for visible light and for mm-wave and THz synthetic aperture radar (SAR).The common feature of the proposed sensors is that they are (a) not designed for generating images, unlike cameras and many existing sensors, and (b) they use differential, contrast-based measurements. For visual light, the new sensors are partially motivated by findings in the human visual system, and they have a potential of being superior to standard cameras in high dynamic range environments. We propose to built sensors that have center-surround receptive fields and opponent-color coding (similar to retinal ganglion cells) and that calculate spatial and temporal derivatives on the analog signal on chip. In the research unit, we also develop and explore foveated sensors, and the focus of this subproject is on differential low-bandwidth data from the periphery of the visual field. For mm-waves and THz radiation, the proposed differential sensors remove substantial parts of the background signal which is due to direct reflections and multipath scattering, giving them the potential to simplify the inverse problems that are involved in computer vision.While many standard cameras and sensors can be optimized by improving on measures of image quality, the non-image-generating sensors can only be optimized and trained in an end-to-end application scenario with specific ML tasks. We explore such settings in the contexts of face and pedestrian detection, optical flow, peripheral vision and localization of structures in mm-wave and THz SAR. The results of this project contribute to the general development of the L2S paradigm, and provide several new and experimental approaches to computer vision.