RI: NSF-BSF: Small: Reconstructing Shape, Lighting and Reflectance Properties of Indoor Scenes from Video

这个项目的目标是允许人们使用室内场景的视频，例如房屋中的一个房间，来重建场景。重建将包含所有场景属性，以使场景的图像能够从新的视点、使用新的照明进行渲染。要做到这一点，必须恢复场景中每个点的深度和形状。还需要描述场景中的每个点如何反射光线，以及恢复场景中的照明。这项技术将在虚拟现实和增强现实中实现广泛的应用。例如，它将帮助生成图像，显示如果家具被更换，房间将是什么样子，或者允许增强现实电话会议，在这种情况下，物理上分开的发言者似乎在同一个房间里聊天。该项目将培养几名研究生。研究人员还将开发相关的本科研究项目，以及为高中生举办的人工智能夏令营项目。这种反向渲染具有挑战性的一个原因是缺乏标记的真实数据。调查人员将通过使用多视角输入来针对场景的所有组件进行训练和联合求解来解决这一缺陷。这将允许他们使用自我监督的多视图光度学重建损失，从而加强视图之间的一致性。他们将把这些真实数据与高度逼真的合成数据的大型新数据集结合起来。他们还将开发一个统一的架构，可以在视图之间恢复姿势，并将几何信息和光度信息结合在一起。这将允许系统预测视点之间的外观变化，并确定测量的外观是否与恢复的照明和反射特性一致。他们还将开发场景中照明的新表示法，以及对象反射这种光的紧凑和有效的方式。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The goal of this project is to allow someone to use a video of an indoor scene, such as a room in a house, to reconstruct the scene. The reconstruction will contain all scene properties needed to enable images of the scene to be rendered from new viewpoints, with new lighting. To do this it is necessary to recover the depth and shape of every point in the scene. It is also necessary to describe how each point in the scene reflects light, and to recover the lighting in the scene. This technology will enable a wide range of applications in virtual and augmented reality. For example, it will assist in generating images that show what a room would look like if the furniture were changed, or to allow augmented reality teleconferencing, in which physically separated speakers appear to chat in the same room. The project will educate several graduate students. The investigators will also develop related undergraduate research projects, and projects for an AI summer camp for high school students.One reason such inverse rendering is challenging is the lack of labeled real data. The investigators will address this deficit by using multi-view input for training and solving jointly for all components of the scene. This will allow them to use a self-supervised multi-view photometric reconstruction loss, enforcing consistency across views. They will combine this real data with a large new dataset of highly realistic synthetic data. They will also develop a unified architecture that recovers pose between views and combines geometric and photometric information. This will allow the system to predict appearance changes across viewpoints and to determine whether measured appearance is consistent with recovered lighting and reflectance properties. They will also develop novel representations of lighting in scenes, and of the way that objects reflect this light that are compact and effective.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.