Neural Signal Representations for Physics-Based Machine Learning and Active 3D Imaging

点击翻译按钮获取中文摘要

In recent years, machine learning techniques have become powerful tools for processing and understanding visual data. However, while these techniques offer new and exciting capabilities, they often fail to generalize outside of their training datasets. On the other hand, many conventional, physics-based modeling pipelines are informed by decades of research and demonstrate robust performance across their input space. The two long term objectives of my research program are: 1. studying and developing a new machine learning framework, known as neural signal representations, that marries the capabilities of neural networks with the robustness of physics-based models. 2. exploiting these capabilities for applications in active 3D imaging, with new methods to recover geometry in challenging conditions: at single photon signal levels and with multiply scattered light. Neural signal representations are an emerging method for representing and optimizing signals; they have recently become popular after demonstrating state-of-the-art performance for multiview 3D reconstruction and neural rendering. The idea is that, rather than storing signals as discrete samples, signals are stored directly in the weights of a neural network. Key advantages of this framework are that it can be flexibly incorporated into physics-based models, the network can learn priors over a space of signals, and large scale, high-dimensional signals can be represented with a far smaller memory footprint compared to conventional arrays. Still, there are many open questions about the behavior of neural representations, and there are serious practical limitations relating to efficiency and scalability. My work will (1) advance the theory and interpretability of neural signal representations, and (2) develop practical and scalable architectures that enable new capabilities in computer vision, physics-based modeling, and 3D reconstruction (Research Aim 1). Active imaging systems, such as lidar (light detection and ranging), have achieved widespread adoption for 3D imaging and scene reconstruction. They capture 3D geometry by emitting a pulse of light and measuring the precise time it takes for light to reflect back from an object. While these systems are used in consumer electronics (iPhone, iPad), self-driving cars, robots, remote sensing systems, and biomedical imaging devices, they have a number of failure modes. For example, current lidar systems fail when the reflected light is too weak or becomes scrambled, such as when imaging at long distances or in situations where the emitted light scatters multiple times before returning. My work will leverage machine learned priors and emerging, single-photon-sensitive detectors to create a new class of 3D imaging systems that (1) recover high-resolution 3D geometry from single photons, and which (2) can recover 3D shape using light that scatters multiple times around corners, behind occluders, or through fog (Research Aim 2).