RI: Small: Understanding and Synthesizing People in 3D Scenes

这个项目推进了基础研究，以理解图像中的人，以及他们如何与环境互动。这个目标很重要，因为在许多涉及图像的真实世界应用程序中，人是至关重要的。例如，车辆和机器人应该感知人，预测他们的行为，并在他们周围安全地操作。在计算机图形学领域，为电影和游戏生成内容的算法应该描绘出外观逼真的人，以及以现实方式与环境互动的人。开发一种足以用于此类应用的对人的计算理解，将至关重要地涉及理解人与他们居住的场景之间的联系。例如，在一个典型的街道场景中，人们往往会出现在人行道或人行横道上行走--但如果突然开始下雨，他们的外表和行为可能会有所不同，例如，带着雨伞，匆忙寻找避难所等。这个项目试图通过以人为中心的新方法来感知世界，建立对场景中的人的计算理解。这项研究还将与教育活动相结合，包括努力扩大对计算机的参与。该项目的技术目标分为两个主要目标，与计算机视觉和计算机图形学目标相对应：(1)以人为中心的图像理解，(2)将人合成为图像。这两个目标都将由新的机器学习方法驱动，这些方法将回答以下一系列问题：给定一个场景的图像，一个人可能出现在哪里，他们会做什么，他们看起来会是什么样子？这些问题不同于典型的计算机视觉任务，即识别人类在图像中的实际位置，而不是涉及对他们可能在哪里的推理。这一推理高度依赖于图像中描述的场景--是否有长凳、下雨等--以及场景的3D几何形状--例如，特定的表面是水平的还是垂直的。因此，该项目将探索人们对场景和3D的学习以及他们与世界的互动。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project advances fundamental research into understanding people in images, and how they interact with their environment. This goal is important because people are crucial in many real-world applications that involve imagery. For example, vehicles and robots should perceive people, predict their behavior, and operate safely around them. In the realm of computer graphics, algorithms that generate content for movies and games should depict people with realistic appearance and who interact with their environment in realistic ways. Developing a computational understanding of people sufficient for such applications will involve, crucially, understanding the connection between people and the scenes they inhabit. For instance, in a typical street scene, people will tend to appear walking on sidewalks or crosswalks -- but if it suddenly starts raining, they might look and act differently, for instance, by carrying umbrellas, hurrying for shelter, etc. This project seeks to build such a computational understanding of people in scenes via new human-centric methods for perceiving the world. The research will also be coupled with educational activities, including efforts to broaden participation in computing.The technical goals of the project fall into two main thrusts corresponding to a computer vision and a computer graphics goal: (1) human-centric understanding of images, and (2) realistic synthesis of people into images. Both of these goals will be driven by new machine learning methods that will answer the following series of questions: given an image of a scene, where might a person appear, what would they be doing, and how would they look? These questions are distinct from the typical computer vision task of identifying where humans actually are in an image, and instead involve reasoning about where they could be. This reasoning is highly dependent on the scene depicted in the image -- are there benches, is it raining, etc. -- and also on the 3D geometry of the scene -- e.g., whether a particular surface is horizontal or vertical. Hence, the project will explore scene- and 3D-aware learning of people and their interactions with the world.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.