CAREER: Active and Action-Centric Visual Understanding

该项目开发了视觉语义规划技术;产生有序的动作序列的问题，这些动作序列将当前世界状态从给定图像或视频中描述的状态改变为查询任务定义的状态。该项目的桥梁之间的差距差距目前的图像理解水平和什么是需要主动了解视觉世界的程度，代理可以计划和执行任务。该项目为识别的关键下一步开发了技术：通过对动作、其前提条件和效果以及视觉规划的语义理解来进行主动和以动作为中心的图像理解。 这样做使几个应用在医疗保健，前瞻性记忆失败护理，视力受损的护理，老年人护理，机器人，娱乐和Education.This研究解决视觉规划问题，需要知道什么动作，他们如何改变世界状态，以及哪些动作序列改变当前状态到一个期望的。成功地主动理解图像需要解决计算机视觉和人工智能交叉点上的几个基本且具有挑战性的问题。该研究的重点是开发一个框架，主动视觉理解，新的可扩展算法的联合检测的行动和他们的论点，新的数据集和表示行动的先决条件和影响，新的算法预测的后果与直观的物理定律的行动，和视觉语义规划。所开发的框架是为主动和以动作为中心的图像理解设计的，通过大规模的语义动作识别，建模动作的前提条件和效果，预测动作的后果，以及视觉规划。这些资源不仅使计算机视觉，机器人和人工智能的新研究方向成为可能，而且还汇集了这些学科的一些独立工作。

This project develops technologies for visual semantic planning; the problem of producing ordered sequences of actions that change the current world state from what is depicted in a given image or video to the state defined by a query task. The project bridges the gap between current levels of image understanding and what is needed to actively understand the visual world to the extent that an agent can plan and perform tasks. The project develops the technology for a crucial next step in recognition: active and action-centric image understanding by semantic understanding of actions, their preconditions and effects, and visual planning. Doing so empowers several applications in healthcare, prospective memory failure care, visually impaired care, elderly care, robotics, entertainment, and education.This research addresses the visual planning problem that entails knowing what actions are, how they change the world state, and which sequences of actions change the current state to a desired one. Successful active understanding of images requires addressing several fundamental and challenging problems at the intersection of computer vision and artificial intelligence. The research is focused on the development of a framework for active visual understanding, new scalable algorithms for joint detection of actions and their arguments, new datasets and representations for actions' preconditions and effects, new algorithms for predicting the consequences of actions with intuitive laws of physics, and visual semantic planning. The developed framework is designed for active and action-centric image understanding by large-scale, semantic action recognition, modeling actions' preconditions and effects, predicting consequences of actions, and visual planning. These resources not only enable new research directions in computer vision, robotics, and AI, but also bring together some of the independent efforts across these disciplines.