Transport-Aware Image Sensors

我们未来五年研究计划的主要目标将是设计，原型和演示一类新的计算图像传感器集成电路，其关键特性是它们是传输感知的。与记录所有入射光的传统图像传感器不同，传输感知成像仪可以根据其在视觉场景中所遵循的实际3D路径，通过编程选择性地检测部分光线。这样的成像仪通过物理阻挡所有“不受欢迎的”光路来获取光传输的选定组件（例如，仅直接贡献，间接贡献，镜面间接贡献等），因此它们不能贡献在传感器上形成的图像。******交通感知相机有望主导机器人和计算机视觉领域的许多高端工业、科学和商业2D和3D成像应用，如增强现实和虚拟现实、自动驾驶汽车、3D打印机和扫描仪、视频游戏、生物医学成像和材料分析。目前广泛用于这些应用的传统摄像机具有非常有限的可编程性，需要应用传输感知成像技术。目前的全功能传输感知相机需要一个大型机械变形数字微镜设备（DMD）来实现可编程传感器屏蔽，这带来了以下障碍：(i)外形尺寸过大-打印机的大小-这是最大的便携式消费电子产品市场的关键障碍；（ii）由于dmd施加的大透镜曲率导致的令人难以接受的失真，（iii）低机电掩模更新速度显着限制了应用范围，（iv）阻碍移动实现的高功耗，以及(v)高成本-在数千美元的范围内。我们将开发一种新型的计算图像传感器，不仅可以消除dmd及其缺点，而且通过电子逐像素掩蔽，还可以提供以前无法实现的编码曝光成像的多功能性。******所提出的图像传感器结合了空间（强度）和时间（相位）曝光编码，并将提供以前无法达到的性能水平，传输感知相机，一个全新的相机类别在他们自己的权利。基于这种图像传感器的相机将提供我们周围世界的无与伦比的视图，其中折射和散射可以选择性地衰减或放大，物体表面可以在具有挑战性的条件下以3D形式重建，远远超出现有最先进成像技术的可能。

The main objective of our research program for the next five-year term will be to design, prototype and demonstrate a new class of computational image sensor integrated circuits whose key property is that they are transport-aware. Unlike conventional image sensors, which record all incident light, transport-aware imagers can be programmed to selectively detect only some of that light, depending on the actual 3D paths it followed through a visual scene. Such an imager acquires selected components of light transport (e.g., direct-only contributions, indirect-only contributions, specular-indirect contributions, etc.) by physically blocking all “undesirable” light paths so they cannot contribute to the image formed on the sensor.****** Transport-aware cameras promise to dominate many high-end industrial, scientific and commercial 2D and 3D imaging applications in robotics and computer vision such as: augmented and virtual reality, self-driving cars, 3D printers and scanners, video games, biomedical imaging, and materials analysis. Conventional cameras which are currently widely used for these applications have very limited programmability needed to apply transport-aware imaging techniques. Current full-functionality transport-aware cameras require a large mechanically deforming digital micromirror device (DMD) to implement programmable sensor masking, which introduces the following handicaps: (i) excessive form factor – the size of a printer – a key barrier to the largest market of portable consumer electronics; (ii) prohibitive distortion due to DMD-imposed large-lens curvature, (iii) low electro-mechanical mask update speed significantly limiting the range of applications, (iv) high power dissipation hindering mobile implementations, and (v) high cost – in the range of thousands of dollars. We will develop a new class of computational image sensors that not only eliminate DMDs and their shortfalls, but, by electronic per-pixel masking, also offer previously unattainable versatility in coded-exposure imaging.****** The proposed image sensors combine spatial (intensity) and temporal (phase) exposure coding and will deliver previously unattainable performance levels to transport-aware cameras, a radically new class of cameras in their own right. Cameras based on such image sensors will offer an unrivaled view of the world around us in which refraction and scattering can be selectively attenuated or amplified and object surfaces can be reconstructed in 3D under challenging conditions well beyond of what is possible with the existing state of the art imaging technologies.