CAREER: Active and Action-Centric Visual Understanding

该项目开发视觉语义规划技术；生成有序动作序列的问题，这些动作将当前世界状态从给定图像或视频中描述的状态更改为查询任务定义的状态。该项目弥合了当前图像理解水平与主动理解视觉世界所需的差距，从而使智能体能够计划和执行任务。该项目为识别的关键下一步开发了技术：通过对动作、前提条件和效果以及视觉规划的语义理解来主动和以动作为中心的图像理解。这样做可以在医疗保健、前瞻性记忆障碍护理、视力受损护理、老年人护理、机器人、娱乐和教育等领域实现多种应用。这项研究解决了视觉规划问题，需要知道什么是动作，它们如何改变世界状态，以及哪些动作序列将当前状态改变为期望状态。成功地主动理解图像需要解决计算机视觉和人工智能交叉的几个基本和具有挑战性的问题。研究的重点是开发一个主动视觉理解框架，用于联合检测动作及其参数的新的可扩展算法，用于动作先决条件和效果的新数据集和表示，用于预测具有直观物理定律的动作后果的新算法，以及视觉语义规划。所开发的框架旨在通过大规模、语义动作识别、动作的先决条件和效果建模、动作后果预测和视觉规划来实现主动和以动作为中心的图像理解。这些资源不仅使计算机视觉，机器人和人工智能的新研究方向成为可能，而且还汇集了这些学科的一些独立成果。

This project develops technologies for visual semantic planning; the problem of producing ordered sequences of actions that change the current world state from what is depicted in a given image or video to the state defined by a query task. The project bridges the gap between current levels of image understanding and what is needed to actively understand the visual world to the extent that an agent can plan and perform tasks. The project develops the technology for a crucial next step in recognition: active and action-centric image understanding by semantic understanding of actions, their preconditions and effects, and visual planning. Doing so empowers several applications in healthcare, prospective memory failure care, visually impaired care, elderly care, robotics, entertainment, and education.This research addresses the visual planning problem that entails knowing what actions are, how they change the world state, and which sequences of actions change the current state to a desired one. Successful active understanding of images requires addressing several fundamental and challenging problems at the intersection of computer vision and artificial intelligence. The research is focused on the development of a framework for active visual understanding, new scalable algorithms for joint detection of actions and their arguments, new datasets and representations for actions' preconditions and effects, new algorithms for predicting the consequences of actions with intuitive laws of physics, and visual semantic planning. The developed framework is designed for active and action-centric image understanding by large-scale, semantic action recognition, modeling actions' preconditions and effects, predicting consequences of actions, and visual planning. These resources not only enable new research directions in computer vision, robotics, and AI, but also bring together some of the independent efforts across these disciplines.