Neural Signal Representations for Physics-Based Machine Learning and Active 3D Imaging

近年来，机器学习技术已经成为处理和理解视觉数据的强大工具。然而，尽管这些技术提供了令人兴奋的新功能，但它们通常无法在训练数据集之外进行泛化。另一方面，许多传统的基于物理的建模管道都经过数十年的研究，并在其输入空间中表现出强大的性能。我的研究计划的两个长期目标是：1。研究和开发一种新的机器学习框架，称为神经信号表示，将神经网络的能力与基于物理模型的鲁棒性结合起来。2.利用这些功能在主动3D成像中的应用，采用新方法在具有挑战性的条件下恢复几何形状：在单光子信号水平和多重散射光下。神经信号表示是一种用于表示和优化信号的新兴方法;在展示了多视图3D重建和神经渲染的最新性能后，它们最近变得流行起来。这个想法是，而不是将信号存储为离散样本，信号直接存储在神经网络的权重中。该框架的主要优点是，它可以灵活地集成到基于物理的模型中，网络可以在信号空间上学习先验知识，并且与传统阵列相比，大规模高维信号可以用更小的内存占用来表示。尽管如此，关于神经表征的行为仍然存在许多悬而未决的问题，并且在效率和可扩展性方面存在严重的实际限制。我的工作将（1）推进神经信号表示的理论和可解释性，（2）开发实用和可扩展的架构，实现计算机视觉，基于物理的建模和3D重建的新功能（研究目标1）。主动成像系统，如激光雷达（光探测和测距），已经实现了3D成像和场景重建的广泛采用。它们通过发射光脉冲并测量光从物体反射回来所需的精确时间来捕捉3D几何形状。虽然这些系统用于消费电子产品（iPhone，iPad），自动驾驶汽车，机器人，遥感系统和生物医学成像设备，但它们具有许多故障模式。例如，当前的激光雷达系统在反射光太弱或变得杂乱时会失效，例如在长距离成像时或在发射光在返回之前多次散射的情况下。我的工作将利用机器学习的先验知识和新兴的单光子敏感探测器来创建一类新的3D成像系统，该系统（1）从单光子恢复高分辨率3D几何形状，并且（2）可以使用多次散射在拐角处，遮挡物后面或通过雾的光来恢复3D形状（研究目标2）。

In recent years, machine learning techniques have become powerful tools for processing and understanding visual data. However, while these techniques offer new and exciting capabilities, they often fail to generalize outside of their training datasets. On the other hand, many conventional, physics-based modeling pipelines are informed by decades of research and demonstrate robust performance across their input space. The two long term objectives of my research program are: 1. studying and developing a new machine learning framework, known as neural signal representations, that marries the capabilities of neural networks with the robustness of physics-based models. 2. exploiting these capabilities for applications in active 3D imaging, with new methods to recover geometry in challenging conditions: at single photon signal levels and with multiply scattered light. Neural signal representations are an emerging method for representing and optimizing signals; they have recently become popular after demonstrating state-of-the-art performance for multiview 3D reconstruction and neural rendering. The idea is that, rather than storing signals as discrete samples, signals are stored directly in the weights of a neural network. Key advantages of this framework are that it can be flexibly incorporated into physics-based models, the network can learn priors over a space of signals, and large scale, high-dimensional signals can be represented with a far smaller memory footprint compared to conventional arrays. Still, there are many open questions about the behavior of neural representations, and there are serious practical limitations relating to efficiency and scalability. My work will (1) advance the theory and interpretability of neural signal representations, and (2) develop practical and scalable architectures that enable new capabilities in computer vision, physics-based modeling, and 3D reconstruction (Research Aim 1). Active imaging systems, such as lidar (light detection and ranging), have achieved widespread adoption for 3D imaging and scene reconstruction. They capture 3D geometry by emitting a pulse of light and measuring the precise time it takes for light to reflect back from an object. While these systems are used in consumer electronics (iPhone, iPad), self-driving cars, robots, remote sensing systems, and biomedical imaging devices, they have a number of failure modes. For example, current lidar systems fail when the reflected light is too weak or becomes scrambled, such as when imaging at long distances or in situations where the emitted light scatters multiple times before returning. My work will leverage machine learned priors and emerging, single-photon-sensitive detectors to create a new class of 3D imaging systems that (1) recover high-resolution 3D geometry from single photons, and which (2) can recover 3D shape using light that scatters multiple times around corners, behind occluders, or through fog (Research Aim 2).