Neural Signal Representations for Physics-Based Machine Learning and Active 3D Imaging

近年来，机器学习技术已经成为处理和理解视觉数据的有力工具。然而，尽管这些技术提供了新的和令人兴奋的能力，但它们往往无法在其训练数据集之外进行推广。另一方面，许多传统的、基于物理的建模管道从数十年的研究中获得了信息，并在其输入空间中展示了稳健的性能。我的研究计划的两个长期目标是：1.研究和开发一种新的机器学习框架，称为神经信号表示法，它将神经网络的能力与基于物理的模型的健壮性结合在一起。2.利用这些能力在主动三维成像中的应用，使用新的方法在具有挑战性的条件下恢复几何图形：在单光子信号水平上和在多个散射光下。神经信号表示是一种表示和优化信号的新兴方法；它们最近在展示了多视角3D重建和神经渲染的最先进性能后变得流行起来。其想法是，信号不是以离散样本的形式存储，而是直接存储在神经网络的权重中。该框架的主要优点是可以灵活地结合到基于物理的模型中，网络可以在信号空间中学习先验知识，并且与传统阵列相比，可以用更小的存储空间来表示大规模、高维信号。尽管如此，关于神经表示的行为仍有许多悬而未决的问题，而且在效率和可扩展性方面也存在严重的实际限制。我的工作将(1)提高神经信号表示的理论和可解释性，以及(2)开发实用和可扩展的体系结构，使其能够在计算机视觉、基于物理的建模和3D重建(研究目标1)方面具有新的能力。主动成像系统，如激光雷达(光探测和测距)，已经在3D成像和场景重建方面获得了广泛的采用。它们通过发射光脉冲并测量光线从物体反射回来的精确时间来捕捉3D几何图形。虽然这些系统用于消费电子产品(iPhone、iPad)、自动驾驶汽车、机器人、遥感系统和生物医学成像设备，但它们有许多故障模式。例如，当前的激光雷达系统在反射光太弱或变得混乱时会失效，例如在长距离成像时，或者在发射的光在返回之前多次散射的情况下。我的工作将利用机器学习的先例和新兴的单光子敏感探测器来创造一种新型的3D成像系统，这种系统(1)从单个光子恢复高分辨率的3D几何图形，并且(2)可以使用在角落、遮光器后面或雾中多次散射的光来恢复3D形状(研究目标2)。

In recent years, machine learning techniques have become powerful tools for processing and understanding visual data. However, while these techniques offer new and exciting capabilities, they often fail to generalize outside of their training datasets. On the other hand, many conventional, physics-based modeling pipelines are informed by decades of research and demonstrate robust performance across their input space. The two long term objectives of my research program are: 1. studying and developing a new machine learning framework, known as neural signal representations, that marries the capabilities of neural networks with the robustness of physics-based models. 2. exploiting these capabilities for applications in active 3D imaging, with new methods to recover geometry in challenging conditions: at single photon signal levels and with multiply scattered light. Neural signal representations are an emerging method for representing and optimizing signals; they have recently become popular after demonstrating state-of-the-art performance for multiview 3D reconstruction and neural rendering. The idea is that, rather than storing signals as discrete samples, signals are stored directly in the weights of a neural network. Key advantages of this framework are that it can be flexibly incorporated into physics-based models, the network can learn priors over a space of signals, and large scale, high-dimensional signals can be represented with a far smaller memory footprint compared to conventional arrays. Still, there are many open questions about the behavior of neural representations, and there are serious practical limitations relating to efficiency and scalability. My work will (1) advance the theory and interpretability of neural signal representations, and (2) develop practical and scalable architectures that enable new capabilities in computer vision, physics-based modeling, and 3D reconstruction (Research Aim 1). Active imaging systems, such as lidar (light detection and ranging), have achieved widespread adoption for 3D imaging and scene reconstruction. They capture 3D geometry by emitting a pulse of light and measuring the precise time it takes for light to reflect back from an object. While these systems are used in consumer electronics (iPhone, iPad), self-driving cars, robots, remote sensing systems, and biomedical imaging devices, they have a number of failure modes. For example, current lidar systems fail when the reflected light is too weak or becomes scrambled, such as when imaging at long distances or in situations where the emitted light scatters multiple times before returning. My work will leverage machine learned priors and emerging, single-photon-sensitive detectors to create a new class of 3D imaging systems that (1) recover high-resolution 3D geometry from single photons, and which (2) can recover 3D shape using light that scatters multiple times around corners, behind occluders, or through fog (Research Aim 2).