Interpretable and Robust L2S - Optimization

在最近机器学习的成功推动下，L2S的目标是建立一个端到端的图像传感和分析系统的学习管道，允许进行下游应用驱动的传感器设计。然而，从机器学习的角度来看，端到端的优化是有希望的，输入传感器数据、神经网络结构和学习的网络权重之间的复杂相互作用放大了几个研究问题，这些问题在一定程度上已经普遍存在于经典的深度学习中。首先，可以预期的是，目前表现良好的体系结构在某种程度上是针对传统的RGB传感器设计而定制的。因此，我们认为传感器的优化应该伴随着神经结构的优化以及网络权重的优化。因此，我们将在一个联合离散优化问题中同时处理传感器和架构优化。联合结构和传感器参数搜索空间上的贝叶斯优化可以有效地寻找最优解。因此，我们将探索不同的传感器布局。为了提高效率，我们将考虑使用图神经网络计算的体系结构和传感器嵌入。第二，关于提高下游任务(如图像分类、语义分割或光流计算)的精度的优化，可能会产生基于训练时可用数据的体系结构/传感器对。然而，与传感器不受端到端优化的传统学习驱动方法相比，即使在轻微的域移动下，这类模型的稳健性预计也会更加脆弱。因此，我们将研究使用正则化和数据增强技术来确保所得到的模型/传感器对具有一定的稳健性的方法。第三，尽管神经网络经常发布高精度的预测，但它们的可解释性通常较低，并且对预测的不确定性的理解非常有限。在解决传感器/机器学习联合优化问题时，这两个问题都得到了加强。一方面，我们将研究如何将现有的网络可视化方法转移到不一定输出类似图像的数据的优化传感器上，这些方法有助于使决策过程更具解释性。另一方面，我们将分析传感器端的局部优化采样如何干扰决策不确定性，例如使用蒙特卡罗退学测量。所有问题都将在传感器模式的背景下考虑，从基于太赫兹测量的优化RGB传感器到光场显微记录，以及需要不同定位精度的应用，例如分类(无定位)、分割(像素精确定位)或光流估计(像素精确定位，亚像素精确映射)。

Driven by the recent successes in machine learning, the goal of L2S is an end-to-end learning pipeline of image sensing and analysis systems which allows a downstream application-driven sensor design.Yet, while an end-to-end optimization is promising from a machine learning perspective, the complex interplay between input sensor data, neural network architecture and learned network weights amplifies several research questions that are to a certain extent, already prevalent in classical deep learning.First, it is to be expected that currently well performing architectures are somewhat tailored to the conventional RGB sensor design. Therefore, we argue that the sensor optimization should be accompanied by an optimization of the neural architecture along with the network weights. We will therefore address both, sensor and architecture optimization, in a joint discrete optimization problem. Bayesian Optimization over a joint architecture and sensor parameter search space can be employed to find optima in an efficient way. We will thereby explore different sensor layouts. For efficiency, we will consider architecture and sensor embeddings computed using graph neural networks.Second, the optimization with respect to improved accuracy on a downstream task such as image classification, semantic segmentation or optical flow computation, might yield architecture/sensor pairs that perform very well on the data available at training time. Yet, the robustness of such models even under slight domain shifts is expected to be more brittle than in conventional learning driven approaches, where the sensor is not subject to the end-to-end optimization. Therefore, we will investigate methods to ensure a certain robustness of the resulting model/sensor pairs using for example regularization and data augmentation techniques.Third, while neural networks often issue highly accurate predictions, their interpretability is usually low and one has very limited understanding of the prediction uncertainty. Both issues are reinforced when addressing a joint sensor/machine learning optimization. On the one hand, we will investigate how existing methods for network visualization, that help to make the decision process more interpretable, can be transferred to optimized sensors that do not necessarily output image-like data. On the other hand, we will analyze how locally optimized sampling on the sensor side interferes with decision uncertainties, measured for example using Monte-Carlo drop-out. All questions will be considered in the context of sensor modalities ranging from optimized RGB sensors over terahertz measurements to lightfield microscopic recordings and with respect to applications that require varying localization precision, such as classification (no localization), segmentation (pixel accurate localization) or optical flow estimation (pixel accurate localizaiton, sub-pixel accurate mapping).