CAREER: Plenoptic Signal Processing --- A Framework for Sampling, Detection, and Estimation using Plenoptic Functions

光与场景中物体的相互作用通常是复杂的。一个图像-只捕获二维空间变化-是很差的装备来解开这些相互作用，并推断一个场景的属性，包括它的形状，反射率，和它的组成。这对于具有尖锐反射、折射和体积散射的场景尤其正确。本研究使用光线及其变换来模拟光与场景的相互作用。这项研究的核心假设是，当使用光线及其变换进行研究时，形状、反射率和材料成分的估计问题往往更简单、更合理。广泛的现实世界对象和场景将受益于这项研究的进展；这包括具有导致相互反射的复杂配置的场景，具有发光，镜面和空间变化反射率的对象，以及透明或半透明的对象。包括机器视觉、显微镜和消费者摄影在内的各种应用都将从这项研究中受益。该项目的教育和推广部分通过相机建造研讨会和面向中学生的实验室演示，以及面向物理教师的专业发展课程，在更广泛的匹兹堡地区传播图像处理研究。研究的重点是通过研究超越图像的光特征来开发新的场景理解获取和处理方法。该研究特别分析了两种信号的特性：捕捉光的空间、时间、角度和光谱变化的全光函数，以及捕捉光如何在场景中传播的全光传输。该研究的中心假设是，全视功能和光传输提供了光与场景相互作用的丰富编码；因此，与基于图像的推理不同，全光学推理即使对于以复杂方式与光相互作用的场景，也可以从根本上得到良好的条件。为此，该研究开发了基于控制光与场景相互作用的物理定律的全光学函数的新型低维模型。该研究还构建了新型计算相机，通过分解成不同复杂性的光路来获取光在场景中的传播，并随后估计3D形状、反射率和材料组成。

The interactions of light with objects in a scene are often complex. An image --- which only captures 2D spatial variations --- is poorly equipped to unravel these interactions and infer properties of a scene including its shape, reflectance, and its composition. This is especially true for scenes that have sharp reflections, refractions, and volumetric scattering. This research models interactions of light with scenes using light rays and their transformations. The central hypothesis underlying the research is the idea that problems of shape, reflectance and material composition estimation are often simpler and well-posed when they are studied using light rays and their transformations. A wide-range of real-world objects and scenes stand to benefit from progress made in this research; this includes scenes with complex configurations that lead to inter-reflections, objects with shine, specularities, and spatially-varying reflectances, as well as objects that are transparent, or translucent. A diverse set of applications including machine vision, microscopy, and consumer photography stand to benefit from this research. The education and outreach components of this project disseminates image processing research in the broader Pittsburgh area via camera building workshops and lab demos for middle/high-school students, and professional development courses for physics teachers.The focus of the research is to develop novel acquisition and processing methods for scene understanding by studying characterizations of light that go beyond images. In particular, the research analyzes the properties of two signals: the plenoptic function, which captures spatial, temporal, angular, and spectral variations of light, and the plenoptic light transport, which captures how light propagates through a scene. The central hypothesis of the research is that the plenoptic function and light transport provide a rich encoding of how light interacts with a scene; hence, unlike image-based inference, plenoptic inference can be fundamentally well-conditioned even for scenes that interact with light in a complex manner. To this end, the research develops novel low-dimensional models for plenoptic functions that are based on physical laws governing interaction of light with a scene. The research also builds novel computational cameras that acquire light propagates in a scene by decomposing into light paths of varying complexity, and subsequently estimating the 3D shape, reflectance, and material composition.