RI: Small: Understanding and Synthesizing People in 3D Scenes

RI：小：理解和合成 3D 场景中的人物

• 批准号: 2008313
• 负责人: Noah Snavely
• 金额: $ 49.86万
• 依托单位: Cornell University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-10-01 至 2024-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2008313&HistoricalAwards=false
• 关键词: RI; Small; Understanding; Synthesizing; People

This project advances fundamental research into understanding people in images, and how they interact with their environment. This goal is important because people are crucial in many real-world applications that involve imagery. For example, vehicles and robots should perceive people, predict their behavior, and operate safely around them. In the realm of computer graphics, algorithms that generate content for movies and games should depict people with realistic appearance and who interact with their environment in realistic ways. Developing a computational understanding of people sufficient for such applications will involve, crucially, understanding the connection between people and the scenes they inhabit. For instance, in a typical street scene, people will tend to appear walking on sidewalks or crosswalks -- but if it suddenly starts raining, they might look and act differently, for instance, by carrying umbrellas, hurrying for shelter, etc. This project seeks to build such a computational understanding of people in scenes via new human-centric methods for perceiving the world. The research will also be coupled with educational activities, including efforts to broaden participation in computing.The technical goals of the project fall into two main thrusts corresponding to a computer vision and a computer graphics goal: (1) human-centric understanding of images, and (2) realistic synthesis of people into images. Both of these goals will be driven by new machine learning methods that will answer the following series of questions: given an image of a scene, where might a person appear, what would they be doing, and how would they look? These questions are distinct from the typical computer vision task of identifying where humans actually are in an image, and instead involve reasoning about where they could be. This reasoning is highly dependent on the scene depicted in the image -- are there benches, is it raining, etc. -- and also on the 3D geometry of the scene -- e.g., whether a particular surface is horizontal or vertical. Hence, the project will explore scene- and 3D-aware learning of people and their interactions with the world.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目推进了理解图像中的人以及他们如何与环境互动的基础研究。这个目标很重要，因为在许多涉及图像的现实世界应用中，人至关重要。例如，车辆和机器人应该感知人，预测他们的行为，并在他们周围安全地运行。在计算机图形学领域，为电影和游戏生成内容的算法应该以逼真的外观描绘人，并以逼真的方式与他们的环境进行交互。发展一种足以满足这些应用的对人的计算理解，关键是要理解人与他们所居住的场景之间的联系。例如，在一个典型的街道场景中，人们往往会出现在人行道或人行横道上行走-但如果突然开始下雨，他们可能会看起来和行为不同，例如，携带雨伞，匆忙躲避等。该研究还将与教育活动相结合，包括努力扩大对计算的参与。该项目的技术目标分为两个主要目标，分别对应于计算机视觉和计算机图形学目标：（1）以人为中心的图像理解，（2）将人逼真地合成为图像。这两个目标都将由新的机器学习方法驱动，这些方法将回答以下一系列问题：给定一个场景的图像，一个人可能会出现在哪里，他们会做什么，他们看起来会是什么样子？这些问题与典型的计算机视觉任务不同，即识别人类在图像中的实际位置，而是涉及推理他们可能在哪里。这种推理高度依赖于图像中描绘的场景--有长凳吗，下雨了吗，等等--以及场景的3D几何形状--例如，无论特定表面是水平的还是垂直的。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

IRON: Inverse Rendering by Optimizing Neural SDFs and Materials from Photometric Images

• DOI: 10.1109/cvpr52688.2022.00548
• 发表时间: 2022-04
• 期刊: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Kai Zhang;Fujun Luan;Zhengqi Li;Noah Snavely

PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting

• DOI: 10.1109/cvpr46437.2021.00541
• 发表时间: 2021-04
• 期刊: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Kai Zhang;Fujun Luan;Qianqian Wang;Kavita Bala;Noah Snavely

Who’s Waldo? Linking People Across Text and Images

• DOI: 10.1109/iccv48922.2021.00141
• 发表时间: 2021-08
• 期刊: 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
• 作者: Claire Yuqing Cui;Apoorv Khandelwal;Yoav Artzi;Noah Snavely;Hadar Averbuch-Elor

Extreme Rotation Estimation using Dense Correlation Volumes

• DOI: 10.1109/cvpr46437.2021.01433
• 发表时间: 2021-04
• 期刊: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Ruojin Cai;Bharath Hariharan;Noah Snavely;Hadar Averbuch-Elor

Towers of Babel: Combining Images, Language, and 3D Geometry for Learning Multimodal Vision

• DOI: 10.1109/iccv48922.2021.00048
• 发表时间: 2021-08
• 期刊: 2021 IEEE/CVF International Conference on Computer Vision (ICCV)
• 作者: Xiaoshi Wu;Hadar Averbuch-Elor;J. Sun;Noah Snavely