CHS: Small: Collaborative Research: Detailed Shape and Reflectance Capture with Light Field Cameras

CHS：小型：协作研究：使用光场相机捕获详细形状和反射率

• 批准号: 1617234
• 负责人: Ravi Ramamoorthi
• 金额: $ 25万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-09-01 至 2020-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1617234&HistoricalAwards=false
• 关键词: CHS; Small; Collaborative; Research; Detailed

A major transformation is occurring in the way we sense the visual world. Traditional 2D photography is increasingly being replaced with light-field sensors that capture the full spatial and angular variation of the incoming light field, rather than simply pixels that integrate over incoming directions. This development opens up the possibility of ubiquitous 3D imaging of our visual world. Light-field sensors are particularly attractive as a depth acquisition device, since they are completely passive without needing to project light into the scene, and they do not experience a reduction in performance outdoors. Moreover, the rich ray-space of light fields provides significant cues for recovering fine-scale depth. However, current RGBD and light-field systems produce only coarse depth; while useful for tasks like refocusing images, the depth channel offers little benefit for photography beyond conventional 2D RGB images. This research seeks to address these challenges, by developing practical algorithms for detailed 3D shape and reflectance capture with light-field cameras, coupled with a theoretical and experimental analysis of the achievable accuracy. Project outcomes will have broad impact on diverse fields including computer graphics and virtual/augmented reality, enabling acquisition of high-quality detailed 3D shape and the subsequent use of the 3D geometry with computer-generated synthetic objects. Methods to acquire 3D images, including on mobile sensors, will transform the photographic process from 2D to 3D, with immense industrial and societal impact.The PIs will address four important problems in light-field shape acquisition. First, they will exploit the rich nature of light-field data, combining multiple cues (defocus, correspondence, shading, specularity) in a unified way to obtain the overall global 3D shape. Moreover, they will seek to go beyond the common Lambertian reflectance assumption, developing a novel BRDF-invariant framework for surface reconstruction with general glossy materials like metals, plastics, or ceramics, while supporting textures and spatially-varying reflectance. Another key objective will be to ground the practical results in a theoretical framework that can establish the limits of light-field camera shape resolution, and the signal-to-noise accuracy, and how this relates to novel designs for light-field cameras to obtain the best achievable resolution in 3D shape capture. Finally, the PIs will move from overall shape to fine-scale surface detail, proposing new methods for shape/reflectance capture for fine-scale geometry like hair. The ultimate goal is to enable a full 3D processing pipeline for photography, computer graphics and applications like virtual and augmented reality.

我们感知视觉世界的方式正在发生重大转变。 传统的2D摄影越来越多地被光场传感器所取代，这些传感器捕捉入射光场的全部空间和角度变化，而不仅仅是在入射方向上集成的像素。 这一发展为我们的视觉世界提供了无处不在的3D成像的可能性。 光场传感器作为深度采集设备特别有吸引力，因为它们是完全无源的，不需要将光投射到场景中，并且它们在户外不会降低性能。 此外，光场丰富的光线空间为恢复精细尺度深度提供了重要线索。 然而，目前的RGBD和光场系统只产生粗略的深度;虽然对于重新聚焦图像等任务很有用，但深度通道对传统2D RGB图像之外的摄影几乎没有好处。 本研究旨在解决这些挑战，通过开发实用的算法，详细的3D形状和反射率捕获与光场相机，再加上理论和实验分析的可实现的精度。 项目成果将对包括计算机图形和虚拟/增强现实在内的各个领域产生广泛影响，从而能够获得高质量的详细3D形状，并随后将3D几何形状与计算机生成的合成对象一起使用。 包括在移动的传感器上获取3D图像的方法将把摄影过程从2D转变为3D，具有巨大的工业和社会影响。 首先，他们将利用光场数据的丰富性质，以统一的方式组合多个线索（散焦，对应，阴影，镜面反射）以获得整体全局3D形状。 此外，他们将寻求超越常见的朗伯反射假设，开发一种新的BRDF不变框架，用于使用金属，塑料或陶瓷等一般光泽材料进行表面重建，同时支持纹理和空间变化的反射率。 另一个关键目标将是在理论框架中建立实际结果，该理论框架可以建立光场相机形状分辨率的限制和信噪比精度，以及这与光场相机的新颖设计如何相关，以获得3D形状捕获中可实现的最佳分辨率。 最后，PI将从整体形状移动到精细尺度表面细节，为精细尺度几何体（如头发）的形状/反射率捕获提出新方法。 最终目标是为摄影，计算机图形和虚拟和增强现实等应用程序提供完整的3D处理管道。