CHS: Small: Collaborative Research: Detailed Shape and Reflectance Capture with Light Field Cameras

我们感知视觉世界的方式正在发生重大转变。传统的2D摄影正逐渐被光场传感器所取代，这种传感器可以捕捉入射光场的全部空间和角度变化，而不是简单地在入射方向上集成像素。这一发展为我们的视觉世界提供了无处不在的3D成像的可能性。光场传感器作为一种深度采集设备特别有吸引力，因为它们是完全被动的，不需要将光投射到场景中，而且它们在户外的性能不会下降。此外，光场丰富的光线空间为精细尺度深度的恢复提供了重要线索。然而，目前的RGBD和光场系统只能产生粗糙的深度；虽然深度通道对于像重新聚焦图像这样的任务很有用，但除了传统的2D RGB图像之外，它对摄影几乎没有什么好处。本研究旨在解决这些挑战，通过开发用于光场相机详细3D形状和反射率捕获的实用算法，以及对可实现精度的理论和实验分析。项目成果将对包括计算机图形学和虚拟/增强现实在内的多个领域产生广泛影响，使获取高质量的详细3D形状以及随后使用计算机生成的合成对象的3D几何形状成为可能。获取3D图像的方法，包括移动传感器，将把摄影过程从2D转变为3D，对工业和社会产生巨大影响。pi将解决光场形状获取中的四个重要问题。首先，他们将利用光场数据的丰富特性，以统一的方式结合多个线索（散焦、对应、阴影、镜面）来获得整体的3D形状。此外，他们将寻求超越常见的朗伯反射率假设，开发一种新的brdf不变框架，用于使用金属、塑料或陶瓷等一般光滑材料进行表面重建，同时支持纹理和空间变化反射率。另一个关键目标将是在理论框架中建立实际结果，该框架可以确定光场相机形状分辨率的限制，以及信噪比精度，以及这与光场相机的新设计如何相关，以便在3D形状捕获中获得最佳可实现的分辨率。最后，pi将从整体形状转移到精细尺度的表面细节，提出精细尺度几何形状（如头发）的形状/反射率捕获新方法。最终目标是为摄影、计算机图形以及虚拟和增强现实等应用程序提供完整的3D处理管道，这将为休闲摄影师带来无处不在的3D。

A major transformation is occurring in the way we sense the visual world. Traditional 2D photography is increasingly being replaced with light-field sensors that capture the full spatial and angular variation of the incoming light field, rather than simply pixels that integrate over incoming directions. This development opens up the possibility of ubiquitous 3D imaging of our visual world. Light-field sensors are particularly attractive as a depth acquisition device, since they are completely passive without needing to project light into the scene, and they do not experience a reduction in performance outdoors. Moreover, the rich ray-space of light fields provides significant cues for recovering fine-scale depth. However, current RGBD and light-field systems produce only coarse depth; while useful for tasks like refocusing images, the depth channel offers little benefit for photography beyond conventional 2D RGB images. This research seeks to address these challenges, by developing practical algorithms for detailed 3D shape and reflectance capture with light-field cameras, coupled with a theoretical and experimental analysis of the achievable accuracy. Project outcomes will have broad impact on diverse fields including computer graphics and virtual/augmented reality, enabling acquisition of high-quality detailed 3D shape and the subsequent use of the 3D geometry with computer-generated synthetic objects. Methods to acquire 3D images, including on mobile sensors, will transform the photographic process from 2D to 3D, with immense industrial and societal impact.The PIs will address four important problems in light-field shape acquisition. First, they will exploit the rich nature of light-field data, combining multiple cues (defocus, correspondence, shading, specularity) in a unified way to obtain the overall global 3D shape. Moreover, they will seek to go beyond the common Lambertian reflectance assumption, developing a novel BRDF-invariant framework for surface reconstruction with general glossy materials like metals, plastics, or ceramics, while supporting textures and spatially-varying reflectance. Another key objective will be to ground the practical results in a theoretical framework that can establish the limits of light-field camera shape resolution, and the signal-to-noise accuracy, and how this relates to novel designs for light-field cameras to obtain the best achievable resolution in 3D shape capture. Finally, the PIs will move from overall shape to fine-scale surface detail, proposing new methods for shape/reflectance capture for fine-scale geometry like hair. The ultimate goal is to enable a full 3D processing pipeline for photography, computer graphics and applications like virtual and augmented reality, which will bring ubiquitous 3D to the casual photographer.