Neural Signal Representations for Physics-Based Machine Learning and Active 3D Imaging

基于物理的机器学习和主动 3D 成像的神经信号表示

• 批准号: RGPIN-2022-04829
• 负责人: Lindell, David
• 金额: $ 2.91万
• 依托单位: University of Toronto
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=750045
• 关键词: Neural; Signal; Representations; Physics; Based

In recent years, machine learning techniques have become powerful tools for processing and understanding visual data. However, while these techniques offer new and exciting capabilities, they often fail to generalize outside of their training datasets. On the other hand, many conventional, physics-based modeling pipelines are informed by decades of research and demonstrate robust performance across their input space. The two long term objectives of my research program are: 1. studying and developing a new machine learning framework, known as neural signal representations, that marries the capabilities of neural networks with the robustness of physics-based models. 2. exploiting these capabilities for applications in active 3D imaging, with new methods to recover geometry in challenging conditions: at single photon signal levels and with multiply scattered light. Neural signal representations are an emerging method for representing and optimizing signals; they have recently become popular after demonstrating state-of-the-art performance for multiview 3D reconstruction and neural rendering. The idea is that, rather than storing signals as discrete samples, signals are stored directly in the weights of a neural network. Key advantages of this framework are that it can be flexibly incorporated into physics-based models, the network can learn priors over a space of signals, and large scale, high-dimensional signals can be represented with a far smaller memory footprint compared to conventional arrays. Still, there are many open questions about the behavior of neural representations, and there are serious practical limitations relating to efficiency and scalability. My work will (1) advance the theory and interpretability of neural signal representations, and (2) develop practical and scalable architectures that enable new capabilities in computer vision, physics-based modeling, and 3D reconstruction (Research Aim 1). Active imaging systems, such as lidar (light detection and ranging), have achieved widespread adoption for 3D imaging and scene reconstruction. They capture 3D geometry by emitting a pulse of light and measuring the precise time it takes for light to reflect back from an object. While these systems are used in consumer electronics (iPhone, iPad), self-driving cars, robots, remote sensing systems, and biomedical imaging devices, they have a number of failure modes. For example, current lidar systems fail when the reflected light is too weak or becomes scrambled, such as when imaging at long distances or in situations where the emitted light scatters multiple times before returning. My work will leverage machine learned priors and emerging, single-photon-sensitive detectors to create a new class of 3D imaging systems that (1) recover high-resolution 3D geometry from single photons, and which (2) can recover 3D shape using light that scatters multiple times around corners, behind occluders, or through fog (Research Aim 2).

近年来，机器学习技术已经成为处理和理解视觉数据的强大工具。然而，尽管这些技术提供了令人兴奋的新功能，但它们往往无法泛化其训练数据集之外的数据。另一方面，许多传统的、基于物理的建模管道经过几十年的研究，在其输入空间中表现出强大的性能。我的研究计划的两个长期目标是：1。研究和开发一种新的机器学习框架，称为神经信号表示，它将神经网络的能力与基于物理模型的鲁棒性相结合。2. 将这些功能应用于主动3D成像，并采用新方法在具有挑战性的条件下恢复几何形状：在单光子信号水平和多重散射光下。神经信号表示是一种新兴的信号表示和优化方法；在展示了最先进的多视图3D重建和神经渲染性能后，它们最近变得流行起来。其思想是，与其将信号存储为离散样本，不如将信号直接存储在神经网络的权重中。该框架的主要优点是它可以灵活地整合到基于物理的模型中，网络可以在信号空间中学习先验，并且与传统阵列相比，大规模，高维信号可以用更小的内存占用来表示。尽管如此，关于神经表征的行为仍有许多悬而未决的问题，并且在效率和可扩展性方面存在严重的实际限制。我的工作将(1)推进神经信号表示的理论和可解释性，(2)开发实用和可扩展的架构，使计算机视觉，基于物理的建模和3D重建的新功能成为可能（研究目标1）。主动成像系统，如激光雷达（光探测和测距），已经广泛应用于3D成像和场景重建。他们通过发射光脉冲并测量光从物体反射回来所需的精确时间来捕捉3D几何形状。虽然这些系统用于消费电子产品（iPhone, iPad），自动驾驶汽车，机器人，遥感系统和生物医学成像设备，但它们有许多故障模式。例如，当前的激光雷达系统在反射光太弱或变得混乱时就会失效，例如在远距离成像或在发射的光在返回之前多次散射的情况下。我的工作将利用机器学习先验和新兴的单光子敏感探测器来创建一类新的3D成像系统，该系统(1)从单光子中恢复高分辨率3D几何形状，并且(2)可以使用在角落周围，遮挡物后面或通过雾散射多次的光来恢复3D形状（研究目标2）。