CHS: Small: Collaborative Research: Detailed Shape and Reflectance Capture with Light Field Cameras

我们感知视觉世界的方式正在发生重大变化。传统的2D摄影越来越多地被光场传感器所取代，这种传感器可以捕捉入射光场的全部空间和角度变化，而不是简单地在入射方向上积分的像素。这一发展为我们的视觉世界提供了无处不在的3D成像的可能性。光场传感器作为一种深度获取设备特别有吸引力，因为它们是完全被动的，不需要将光线投射到场景中，而且它们不会在户外体验性能下降。此外，丰富的光场射线空间为恢复精细尺度深度提供了重要线索。然而，目前的RGBD和光场系统只能产生粗略的深度；虽然深度通道对于重新对焦图像等任务很有用，但除了传统的2D RGB图像外，深度通道对摄影几乎没有什么好处。这项研究试图解决这些挑战，开发实用的算法，用光场相机捕捉详细的3D形状和反射比，并对可实现的精度进行理论和实验分析。项目成果将对包括计算机图形学和虚拟/增强现实在内的不同领域产生广泛影响，从而能够获得高质量的详细3D形状，并随后将3D几何图形与计算机生成的合成对象一起使用。获取3D图像的方法，包括在移动传感器上，将把摄影过程从2D转变为3D，具有巨大的工业和社会影响。PI将解决光场形状获取中的四个重要问题。首先，他们将利用光场数据的丰富性质，以统一的方式组合多个线索(散焦、对应、阴影、镜面反射)，以获得整体的3D形状。此外，他们将寻求超越常见的朗伯反射比假设，开发一种新的BRDF不变框架，用于使用普通光滑材料(如金属、塑料或陶瓷)重建表面，同时支持纹理和空间变化的反射率。另一个关键目标将是将实际结果建立在一个理论框架中，该框架可以确定光场相机形状分辨率和信噪比精度的限制，以及这与光场相机的新设计如何关联，以在3D形状捕获中获得最佳可实现的分辨率。最后，PI将从整体形状转移到精细曲面细节，为头发等精细几何图形提供形状/反射比捕捉的新方法。最终目标是为摄影、计算机图形以及虚拟和增强现实等应用程序提供完整的3D处理流水线，这将为休闲摄影师带来无处不在的3D。

A major transformation is occurring in the way we sense the visual world. Traditional 2D photography is increasingly being replaced with light-field sensors that capture the full spatial and angular variation of the incoming light field, rather than simply pixels that integrate over incoming directions. This development opens up the possibility of ubiquitous 3D imaging of our visual world. Light-field sensors are particularly attractive as a depth acquisition device, since they are completely passive without needing to project light into the scene, and they do not experience a reduction in performance outdoors. Moreover, the rich ray-space of light fields provides significant cues for recovering fine-scale depth. However, current RGBD and light-field systems produce only coarse depth; while useful for tasks like refocusing images, the depth channel offers little benefit for photography beyond conventional 2D RGB images. This research seeks to address these challenges, by developing practical algorithms for detailed 3D shape and reflectance capture with light-field cameras, coupled with a theoretical and experimental analysis of the achievable accuracy. Project outcomes will have broad impact on diverse fields including computer graphics and virtual/augmented reality, enabling acquisition of high-quality detailed 3D shape and the subsequent use of the 3D geometry with computer-generated synthetic objects. Methods to acquire 3D images, including on mobile sensors, will transform the photographic process from 2D to 3D, with immense industrial and societal impact.The PIs will address four important problems in light-field shape acquisition. First, they will exploit the rich nature of light-field data, combining multiple cues (defocus, correspondence, shading, specularity) in a unified way to obtain the overall global 3D shape. Moreover, they will seek to go beyond the common Lambertian reflectance assumption, developing a novel BRDF-invariant framework for surface reconstruction with general glossy materials like metals, plastics, or ceramics, while supporting textures and spatially-varying reflectance. Another key objective will be to ground the practical results in a theoretical framework that can establish the limits of light-field camera shape resolution, and the signal-to-noise accuracy, and how this relates to novel designs for light-field cameras to obtain the best achievable resolution in 3D shape capture. Finally, the PIs will move from overall shape to fine-scale surface detail, proposing new methods for shape/reflectance capture for fine-scale geometry like hair. The ultimate goal is to enable a full 3D processing pipeline for photography, computer graphics and applications like virtual and augmented reality, which will bring ubiquitous 3D to the casual photographer.