RI: NSF-BSF: Small: Reconstructing Shape, Lighting and Reflectance Properties of Indoor Scenes from Video

这个项目的目标是允许人们使用室内场景的视频，比如房子里的一个房间，来重建场景。 重建将包含所有场景属性，使场景的图像能够从新的视点渲染，新的照明。 要做到这一点，有必要恢复场景中每个点的深度和形状。 还需要描述场景中的每个点如何反射光线，并恢复场景中的照明。 这项技术将在虚拟和增强现实中实现广泛的应用。 例如，它将有助于生成图像，显示如果家具被改变，房间会是什么样子，或者允许增强现实电话会议，其中物理上分离的扬声器似乎在同一个房间里聊天。 该项目将培养几名研究生。 研究人员还将开发相关的本科研究项目，以及高中生人工智能夏令营的项目。这种逆渲染具有挑战性的一个原因是缺乏标记的真实的数据。调查人员将通过使用多视图输入进行训练和联合解决场景的所有组件来解决这一缺陷。这将允许他们使用自我监督的多视图光度重建损失，强制执行跨视图的一致性。他们将把这些真实数据与一个由高度真实的合成数据组成的大型新数据集结合起来。 他们还将开发一个统一的架构，恢复视图之间的姿态，并结合几何和光度信息。这将允许系统预测视点之间的外观变化，并确定测量的外观是否与恢复的照明和反射特性一致。 他们还将开发照明在场景中的新颖表现，以及物体反射这种光线的方式，这些方式紧凑而有效。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The goal of this project is to allow someone to use a video of an indoor scene, such as a room in a house, to reconstruct the scene. The reconstruction will contain all scene properties needed to enable images of the scene to be rendered from new viewpoints, with new lighting. To do this it is necessary to recover the depth and shape of every point in the scene. It is also necessary to describe how each point in the scene reflects light, and to recover the lighting in the scene. This technology will enable a wide range of applications in virtual and augmented reality. For example, it will assist in generating images that show what a room would look like if the furniture were changed, or to allow augmented reality teleconferencing, in which physically separated speakers appear to chat in the same room. The project will educate several graduate students. The investigators will also develop related undergraduate research projects, and projects for an AI summer camp for high school students.One reason such inverse rendering is challenging is the lack of labeled real data. The investigators will address this deficit by using multi-view input for training and solving jointly for all components of the scene. This will allow them to use a self-supervised multi-view photometric reconstruction loss, enforcing consistency across views. They will combine this real data with a large new dataset of highly realistic synthetic data. They will also develop a unified architecture that recovers pose between views and combines geometric and photometric information. This will allow the system to predict appearance changes across viewpoints and to determine whether measured appearance is consistent with recovered lighting and reflectance properties. They will also develop novel representations of lighting in scenes, and of the way that objects reflect this light that are compact and effective.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.