CAREER: Plenoptic Signal Processing --- A Framework for Sampling, Detection, and Estimation using Plenoptic Functions

灯光与场景中对象的交互通常很复杂。一幅图像-只捕捉2D空间变化-不足以解开这些相互作用，并推断场景的属性，包括其形状、反射率和组成。对于具有尖锐反射、折射和体积散射的场景尤其如此。这项研究使用光线及其变换来模拟光线与场景的相互作用。这项研究背后的中心假设是，当使用光线及其变换来研究形状、反射率和材料成分估计时，它们的问题往往更简单、更合适。广泛的真实世界对象和场景将从这项研究中取得的进展中受益；这包括具有导致相互反射的复杂配置的场景、具有光泽、镜面反射率和空间变化的反射率的对象，以及透明或半透明的对象。包括机器视觉、显微镜和消费者摄影在内的各种应用都将从这项研究中受益。该项目的教育和推广部分通过为初中生和高中生举办的相机制造研讨会和实验室演示，以及为物理教师提供的专业发展课程，在更广泛的匹兹堡地区传播图像处理研究。研究的重点是通过研究超出图像的光的特征，开发用于场景理解的新的获取和处理方法。特别是，这项研究分析了两种信号的特性：全光函数，捕获光的空间、时间、角度和光谱变化，以及全光传输，捕获光如何通过场景传播。这项研究的中心假设是，全光功能和光传输为光如何与场景相互作用提供了丰富的编码；因此，与基于图像的推理不同，全光推理即使对于以复杂方式与光相互作用的场景，也可以从根本上良好地进行。为此，该研究开发了全光函数的新型低维模型，该模型基于控制光与场景相互作用的物理定律。这项研究还建立了新型的计算相机，通过将光分解成不同复杂程度的光路，然后估计3D形状、反射率和材料组成，来获取光在场景中的传播。

The interactions of light with objects in a scene are often complex. An image --- which only captures 2D spatial variations --- is poorly equipped to unravel these interactions and infer properties of a scene including its shape, reflectance, and its composition. This is especially true for scenes that have sharp reflections, refractions, and volumetric scattering. This research models interactions of light with scenes using light rays and their transformations. The central hypothesis underlying the research is the idea that problems of shape, reflectance and material composition estimation are often simpler and well-posed when they are studied using light rays and their transformations. A wide-range of real-world objects and scenes stand to benefit from progress made in this research; this includes scenes with complex configurations that lead to inter-reflections, objects with shine, specularities, and spatially-varying reflectances, as well as objects that are transparent, or translucent. A diverse set of applications including machine vision, microscopy, and consumer photography stand to benefit from this research. The education and outreach components of this project disseminates image processing research in the broader Pittsburgh area via camera building workshops and lab demos for middle/high-school students, and professional development courses for physics teachers.The focus of the research is to develop novel acquisition and processing methods for scene understanding by studying characterizations of light that go beyond images. In particular, the research analyzes the properties of two signals: the plenoptic function, which captures spatial, temporal, angular, and spectral variations of light, and the plenoptic light transport, which captures how light propagates through a scene. The central hypothesis of the research is that the plenoptic function and light transport provide a rich encoding of how light interacts with a scene; hence, unlike image-based inference, plenoptic inference can be fundamentally well-conditioned even for scenes that interact with light in a complex manner. To this end, the research develops novel low-dimensional models for plenoptic functions that are based on physical laws governing interaction of light with a scene. The research also builds novel computational cameras that acquire light propagates in a scene by decomposing into light paths of varying complexity, and subsequently estimating the 3D shape, reflectance, and material composition.