Transport-Aware Imaging for Computer Vision

我未来五年的研究计划的主要目标将是研究、设计和使用一种新的计算相机，其关键特性是它们是“交通感知”的。与记录所有入射光线的传统相机不同，交通感知相机可以根据场景中实际的3D路径，对其进行编程，以阻挡部分光线。交通感知相机使用可编程光源照明和可编程传感器掩模成像，这是三年前由我的研究小组率先开发的。 来自交通感知摄像头的实时视频可以为我们的日常世界提供非常非传统的视角，在这个世界中，折射和散射可以被选择性地阻挡或增强，肉眼无法察觉的视觉结构可以变得明显，物体表面可以在远远超出技术水平的具有挑战性的条件下以3D方式重建。我们在利用它们的能力方面的初步工作结合了计算机视觉、计算机图形学、数值优化和几何光学，以展示前所未有的能力，在3D形状获取、稳健的飞行时间成像、视觉材料分析、场景理解、机器人导航、工业检查和医疗/科学成像等方面具有潜在的应用。 我们的计划试图通过(1)推广交通感知成像的概念并推进其数学基础；(2)设计和构建新颖的交通感知相机系统；以及(3)利用这些新发现的视觉输入来源来解决具有挑战性的3D和2D视觉推理任务，从而释放这些相机的全部潜力。

The main goal of my research program for the next five years will be to study, design, and use a new class of computational cameras whose key property is that they are "transport-aware." Unlike conventional cameras which record all incident light, transport-aware cameras can be programmed to block some of that light, based on the actual 3D paths it followed through a scene. Transport-aware cameras use a programmable light source for illumination and a programmable sensor mask for imaging, and were pioneered by my research group three years ago. Live video from a transport-aware camera can offer a very unconventional view of our everyday world in which refraction and scattering can be selectively blocked or enhanced, visual structures too subtle to notice with the naked eye can become apparent, and object surfaces can be reconstructed in 3D under challenging conditions well beyond the state of the art. Our initial work on harnessing their power brings together computer vision, computer graphics, numerical optimization and geometric optics to demonstrate never-seen-before capabilities with potential applications in 3D shape acquisition; robust time-of-flight imaging; visual material analysis; scene understanding; robotic navigation; industrial inspection; and medical/scientific imaging among others. Our program seeks to unlock the full potential of these cameras by (1) generalizing the concept of transport-aware imaging and advancing its mathematical underpinnings; (2) designing and building novel transport-aware camera systems; and (3) exploiting these newfound sources of visual input to solve challenging 3D and 2D visual inference tasks.