RI: Small: Physically-Based Learning for Shape, Lighting and Material in Complex Indoor Scenes

室内场景涉及可见光和不可见光光源与空间变化的材料和形状相互作用，以及相互反射和阴影，以产生显示复杂局部变化的图像。增强现实等应用需要编辑室内场景以插入新对象，更改场景某些部分的材质，或在不同照明下可视化房间。估计图像形成的这些因素构成了逆绘制的问题，当输入仅是用商品相机获取的室内场景的几幅图像时，逆绘制的问题尤其不适定。虽然传统的基于测量的方法需要昂贵的设备并且不能扩展到大规模场景，但传统的学习方法缺乏处理上述复杂视觉效果的数据和表达能力。这项研究通过对复杂材料和照明进行建模的整体方法，开发新的物理启发的深度网络并生成大规模训练数据，解决了逆向渲染的长期计算机视觉和图形挑战。该项目将通过逼真的增强现实和图像编辑应用程序，如视图合成，对象插入，材料和光源编辑，以及精确的阴影和相互反射，对工业和社会产生变革性影响。该项目还通过计算机视觉和计算机图形学的新跨学科课程，将上述技术进步的经验见解融入大学和K-12教育。该项目开发基于物理的深度网络，结合图像形成的归纳偏差，以实现可概括的表示，以前所未有的细节估计形状，材料，特别是照明，通过考虑单个图像中的亮度、多个图像中的一致性以及具有挑战性或动态场景中的复杂光传输。它设计了新的模块，如神经渲染层和反射体积，结合了图像形成的领域知识。它通过保留描述能力的简约表示实现可扩展性，同时推广到复杂的照明效果，如折射，散射和动态光传输。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Indoor scenes involve visible and invisible light sources interacting with spatially-varying materials and shapes, together with interreflections and shadows, to produce images that display complex local variations. Applications such as augmented reality require editing an indoor scene to insert new objects, changing materials in some parts of the scene, or visualizing a room under a different illumination. Estimating those factors of image formation constitutes the problem of inverse rendering, which is especially ill-posed when the input is only a few images of an indoor scene acquired with a commodity camera. While traditional measurement-based methods need expensive devices and do not scale to large-scale scenes, conventional learning methods suffer from a lack of data and expressive power to handle the above complex visual effects. This research addresses the longstanding computer vision and graphics challenge of inverse rendering through a holistic approach of modeling complex materials and illumination, developing novel physically inspired deep networks and generating large-scale training data. The project will have a transformative effect on industry and society through photorealistic augmented reality and image editing applications such as view synthesis, object insertion, material, and light source editing, with accurate shadows and interreflections. The project also incorporates experiential insights from the above technological advances into college and K-12 education through a new interdisciplinary curriculum in computer vision and computer graphics.This project develops physically based deep networks that incorporate the inductive bias of image formation to achieve generalizable representations that estimate shape, material and especially lighting with unprecedented detail, by accounting for visibilities in a single image, consistency in multiple images and complex light transport in challenging or dynamic scenes. It designs novel modules such as neural rendering layers and reflectance volumes that incorporate the domain knowledge of image formation. It achieves scalability through parsimonious representations that retain descriptive power while generalizing to complex lighting effects such as refraction, scattering and dynamic light transport. It develops large-scale training datasets with realistic spatially varying lighting and complex high-quality material.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.