RI: NSF-BSF: Small: Reconstructing Shape, Lighting and Reflectance Properties of Indoor Scenes from Video

RI：NSF-BSF：小型：从视频重建室内场景的形状、照明和反射率属性

• 批准号: 1910132
• 负责人: David Jacobs
• 金额: $ 49.33万
• 依托单位: University of Maryland, College Park
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-08-15 至 2023-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1910132&HistoricalAwards=false
• 关键词: RI; NSF; BSF; Small; Reconstructing

The goal of this project is to allow someone to use a video of an indoor scene, such as a room in a house, to reconstruct the scene. The reconstruction will contain all scene properties needed to enable images of the scene to be rendered from new viewpoints, with new lighting. To do this it is necessary to recover the depth and shape of every point in the scene. It is also necessary to describe how each point in the scene reflects light, and to recover the lighting in the scene. This technology will enable a wide range of applications in virtual and augmented reality. For example, it will assist in generating images that show what a room would look like if the furniture were changed, or to allow augmented reality teleconferencing, in which physically separated speakers appear to chat in the same room. The project will educate several graduate students. The investigators will also develop related undergraduate research projects, and projects for an AI summer camp for high school students.One reason such inverse rendering is challenging is the lack of labeled real data. The investigators will address this deficit by using multi-view input for training and solving jointly for all components of the scene. This will allow them to use a self-supervised multi-view photometric reconstruction loss, enforcing consistency across views. They will combine this real data with a large new dataset of highly realistic synthetic data. They will also develop a unified architecture that recovers pose between views and combines geometric and photometric information. This will allow the system to predict appearance changes across viewpoints and to determine whether measured appearance is consistent with recovered lighting and reflectance properties. They will also develop novel representations of lighting in scenes, and of the way that objects reflect this light that are compact and effective.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

这个项目的目标是让人们使用室内场景的视频，比如房子里的一个房间，来重建场景。重建将包含所有需要的场景属性，使场景的图像能够从新的视点渲染，使用新的照明。要做到这一点，有必要恢复场景中每个点的深度和形状。还需要描述场景中每个点如何反射光，并恢复场景中的照明。这项技术将在虚拟现实和增强现实中实现广泛的应用。例如，它将帮助生成图像，显示房间的家具被改变后的样子，或者允许增强现实电话会议，在物理上分开的扬声器似乎在同一个房间里聊天。这个项目将培养几名研究生。研究人员还将开发相关的本科生研究项目和高中生人工智能夏令营项目。这种反向呈现具有挑战性的一个原因是缺乏标记的真实数据。调查人员将通过使用多视图输入进行训练和联合解决场景的所有组件来解决这一缺陷。这将允许他们使用自我监督的多视图光度重建损失，强制视图之间的一致性。他们将把这些真实的数据与一个由高度真实的合成数据组成的大型新数据集结合起来。他们还将开发一个统一的架构，可以恢复视图之间的姿态，并结合几何和光度信息。这将允许系统预测不同视点的外观变化，并确定测量的外观是否与恢复的照明和反射特性一致。他们还将开发场景中照明的新表示，以及物体反射光线的方式，这些方式紧凑而有效。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

LD-ZNet: A Latent Diffusion Approach for Text-Based Image Segmentation

• DOI: 10.1109/iccv51070.2023.00384
• 发表时间: 2023-03
• 期刊: 2023 IEEE/CVF International Conference on Computer Vision (ICCV)
• 作者: K. Pnvr;Bharat Singh;P. Ghosh;Behjat Siddiquie;David Jacobs

Object-Aware Cropping for Self-Supervised Learning

• 发表时间: 2021-12
• 期刊: ArXiv
• 作者: Shlok Kumar Mishra;Anshul B. Shah;Ankan Bansal;Abhyuday N. Jagannatha;Abhishek Sharma;David Jacobs;Dilip Krishnan

HaLP: Hallucinating Latent Positives for Skeleton-based Self-Supervised Learning of Actions

• DOI: 10.1109/cvpr52729.2023.01807
• 发表时间: 2023-04
• 期刊: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Anshul B. Shah;A. Roy;Ketul Shah;Shlok Kumar Mishra;David W. Jacobs;A. Cherian;Ramalingam Chellappa

Fast Light-Weight Near-Field Photometric Stereo

• DOI: 10.1109/cvpr52688.2022.01228
• 发表时间: 2022-03
• 期刊: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Daniel Lichy;Soumyadip Sengupta;D. Jacobs

Hyperbolic Contrastive Learning for Visual Representations beyond Objects

• DOI: 10.1109/cvpr52729.2023.00661
• 发表时间: 2022-12
• 期刊: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Songwei Ge;Shlok Kumar Mishra;Simon Kornblith;Chun-Liang Li;David Jacobs