CAREER: Learning to Anticipate with Visual Simulation

职业：学习通过视觉模拟进行预测

• 批准号: 2045586
• 负责人: Alexander Schwing
• 金额: $ 55万
• 依托单位: University of Illinois at Urbana-Champaign
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-09-01 至 2026-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2045586&HistoricalAwards=false
• 关键词: CAREER; Learning; Anticipate; Visual; Simulation

The goal of this research project is to develop methods which are able to accurately forecast what an observed scene looks like a few seconds from now. For instance, given frames of a video showing a traffic intersection, how can a machine anticipate the situation at the intersection a few seconds after the last observed video frame? Humans have a remarkable ability to address this task, which is used permanently, e.g., to safely navigate at an intersection, to effectively collaborate in a kitchen, and even when reading. To address this task, neuroscience hypothesizes that the situation is simulated using mental models. This helps to quickly converge to a set of likely outcomes while ruling out implausible situations. In contrast, present-day computer vision, machine learning and autonomous systems which address this challenge are at their infancy. While the last decade has shown tremendous progress for tasks which analyze observations, e.g., to detect visible objects and segment their contours, present-day systems are challenged when reasoning about something that is not directly observed, e.g., the situation a few seconds from now. Reasoning about the unobserved is challenging because the number of possibilities grows quickly. Yet, the ability to forecast is important for any system that wants to interact safely with its surroundings. To close this gap and lay the foundations for systems to anticipate, this project studies three aspects: 1) representations of the data which are suitable for forecasting, 2) properties of methods that permit accurate forecasting, and 3) what data is necessary to develop accurate models for forecasting.Technically, to address the aforementioned three aspects, the project develops methods which learn how to anticipate via visual simulation. Specifically, the methods use the observed data to retrieve a model of the scene either explicitly or implicitly (Thrust 1). The methods also learn from data how this model is transformed to match likely futures, i.e., the systems learn to perform visual simulation. For this, the methods disentangle geometry, dynamics and relations between observed entities via latent variables (Thrust 2). Disentangling is important because geometry, dynamics and relations influence futures differently. The amount and detail of the annotated data which is used to develop these methods will affect the outcomes. This project studies those relations by collecting a novel dataset (Thrust 3). The representations, algorithms and data innovations will be incorporated into undergraduate and graduate courses as well as an outreach program which is developed to teach audience-centric presentations to undergraduate and graduate students, providing an opportunity to learn to anticipate audience behavior (Thrust 4).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

这个研究项目的目标是开发能够准确预测几秒钟后观察到的场景的方法。例如，给定一段显示交通十字路口的视频帧，机器如何在最后一帧观察到的视频帧之后几秒钟预测十字路口的情况？人类有一种非凡的能力来完成这项任务，这种能力被永久地使用，例如，在十字路口安全地导航，在厨房里有效地合作，甚至在阅读时。为了解决这个问题，神经科学假设这种情况是用心理模型模拟的。这有助于迅速收敛于一组可能的结果，同时排除不合理的情况。相比之下，当今解决这一挑战的计算机视觉、机器学习和自主系统还处于起步阶段。虽然过去十年在分析观察结果的任务方面取得了巨大进展，例如，检测可见物体并分割其轮廓，但当前的系统在对未直接观察到的事物进行推理时面临挑战，例如，几秒钟后的情况。对未观察到的事物进行推理是具有挑战性的，因为可能性的数量增长得很快。然而，对于任何想要与周围环境安全互动的系统来说，预测能力都很重要。为了缩小这一差距并为系统预测奠定基础，本项目研究了三个方面：1)适合预测的数据表示，2)允许准确预测的方法的性质，以及3)开发准确预测模型所需的数据。从技术上讲，为了解决上述三个方面，该项目开发了学习如何通过视觉模拟进行预测的方法。具体来说，这些方法使用观测到的数据来显式或隐式地检索场景模型（Thrust 1）。该方法还从数据中学习如何将该模型转换为匹配可能的未来，即系统学习执行视觉模拟。为此，这些方法通过潜在变量解开了几何、动力学和观察实体之间的关系（Thrust 2）。解纠缠很重要，因为几何、动力学和关系对未来的影响是不同的。用于开发这些方法的注释数据的数量和细节将影响结果。本项目通过收集一个新的数据集（Thrust 3）来研究这些关系。表示、算法和数据创新将被纳入本科和研究生课程，以及一个拓展计划，该计划旨在向本科生和研究生教授以观众为中心的演讲，提供学习预测观众行为的机会（推力4）。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Occupancy Planes for Single-view RGB-D Human Reconstruction

• DOI: 10.48550/arxiv.2208.02817
• 发表时间: 2022-08
• 期刊: ArXiv
• 作者: Xiaoming Zhao;Yuan-Ting Hu;Zhongzheng Ren;A. Schwing

Context-Aware Relative Object Queries to Unify Video Instance and Panoptic Segmentation

• DOI: 10.1109/cvpr52729.2023.00617
• 发表时间: 2023-06
• 期刊: 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Anwesa Choudhuri;Girish V. Chowdhary;A. Schwing

DigGAN: Discriminator gradIent Gap Regularization for GAN Training with Limited Data

• DOI: 10.48550/arxiv.2211.14694
• 发表时间: 2022-11
• 期刊: ArXiv
• 作者: Tiantian Fang;Ruoyu Sun;A. Schwing

Learning to Decompose Visual Features with Latent Textual Prompts

• DOI: 10.48550/arxiv.2210.04287
• 发表时间: 2022-10
• 期刊: ArXiv
• 作者: Feng Wang;Manling Li;Xudong Lin;Hairong Lv;A. Schwing;Heng Ji

CEIP: Combining Explicit and Implicit Priors for Reinforcement Learning with Demonstrations

• DOI: 10.48550/arxiv.2210.09496
• 发表时间: 2022-10
• 期刊: ArXiv
• 作者: Kai Yan;A. Schwing;Yu-Xiong Wang