RI: Small: A Unified Compositional Model for Explainable Video-based Human Activity Parsing

计算机视觉的最终目标是从图像和视频中理解场景和活动。这项任务涉及到不同语义层面的许多知觉和认知过程。视觉分类的下一步是视觉解释，即通过视觉推理和推理来解释视觉实体之间的关系。由于这个问题在不同情况下的巨大变异性，解释视觉场景和活动的语义解析是非常具有挑战性的。这个项目研究如何利用视觉实体的结构组成来克服视觉场景和活动中的多样性。它促进和丰富了计算机视觉的基础研究，并对许多融合应用产生了重大影响，包括自主或辅助驾驶、智能机器人和智能视频监控。这项研究还通过课程开发、学生培训和知识传播为教育做出贡献。它包括与K-12学生的互动，以获得参与和研究机会。本研究旨在开发一种统一的视觉合成模型，它能够以可扩展的端到端方式有效地学习复杂的语义概念，同时实现良好的概括性，并提供视觉数据的可解释性解析。该项目的重点是：(1)原则性模型及其理论基础，通过设计基于概率和/或图的随机语法来模拟结构成分；(2)通过利用数据驱动的模式挖掘来发现结构成分并探索模式如何自我形成，为学习和解析提供有效的计算方法；(3)通过推断人类动作、身体运动和与环境相互作用的复杂成分，对视频人类活动的解析和解释进行可靠的案例研究；和(4)用于人类关节身体姿势估计、上下文对象发现和基于视频的人类活动分析和解释的工具和原型系统。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

An ultimate goal of computer vision is understanding scene and activities from images and video. This task involves many perceptual and cognitive processes at various semantic levels. A next step beyond visual classification is visual interpretation, that is, to explain the relations among visual entities through visual inference and reasoning. Due to the enormous variability across instances of this problem, semantic parsing for explaining a visual scene and activities is highly challenging. This project studies how the structural composition of visual entities can be used to overcome the diversity in the visual scene and activities. It advances and enrich the basic research of computer vision, and brings significant impact on many merging applications, including autonomous or assisted driving, intelligent robots, and intelligent video surveillance. This research also contributes to education through curriculum development, student training, and knowledge dissemination. It includes interactions with K-12 students for participation and research opportunities. This research is to develop a unified visual compositional model that can effectively learn complex semantic concepts in a scalable end-to-end fashion, while achieving good generalizability and providing explainable parsing of the visual data. The project is focused on: (1) a principled model and its theoretical foundation, by designing a stochastic grammar based on the probabilistic And/Or-Graph to model the structural composition; (2) an effective computational approach for learning and parsing, by exploiting data-driven pattern mining to discover structural components and by exploring how the patterns may be self-formed; (3) a solid case study on video human activity parsing and interpretation, by inferring the complex compositions of human actions, body movements, and interaction with the environment; and (4) tools and prototype systems for human articulated body pose estimation, contextual object discovery, and video-based human activity analysis and interpretation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.