RI: Small: A Unified Compositional Model for Explainable Video-based Human Activity Parsing

计算机视觉的最终目标是从图像和视频中理解场景和活动。这一任务涉及到许多不同语义层次的知觉和认知过程。视觉分类的下一步是视觉解释，即通过视觉推理和推理来解释视觉实体之间的关系。由于该问题的实例之间存在巨大的差异，用于解释视觉场景和活动的语义解析具有高度挑战性。本项目研究如何利用视觉实体的结构组合来克服视觉场景和活动的多样性。它推动和丰富了计算机视觉的基础研究，并对许多融合应用产生了重大影响，包括自动驾驶或辅助驾驶，智能机器人和智能视频监控。这项研究也有助于通过课程开发，学生培训和知识传播的教育。它包括与K-12学生的参与和研究机会的互动。 这项研究是开发一个统一的视觉组合模型，可以有效地学习复杂的语义概念，在一个可扩展的端到端的方式，同时实现良好的泛化能力，并提供可解释的分析的视觉数据。该项目的重点是：（1）通过设计一种基于概率与或图的随机文法来模拟结构组成，建立了一个原则模型及其理论基础：（2）通过利用数据驱动的模式挖掘来发现结构成分，并探索模式是如何自形成的，建立了一种有效的学习和分析计算方法;（3）通过推断人类动作、身体运动以及与环境的交互的复杂组成，对视频人类活动解析和解释进行了坚实的案例研究;以及（4）用于人类关节式身体姿态估计，上下文对象发现，视频-该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值进行评估来支持和更广泛的影响审查标准。

An ultimate goal of computer vision is understanding scene and activities from images and video. This task involves many perceptual and cognitive processes at various semantic levels. A next step beyond visual classification is visual interpretation, that is, to explain the relations among visual entities through visual inference and reasoning. Due to the enormous variability across instances of this problem, semantic parsing for explaining a visual scene and activities is highly challenging. This project studies how the structural composition of visual entities can be used to overcome the diversity in the visual scene and activities. It advances and enrich the basic research of computer vision, and brings significant impact on many merging applications, including autonomous or assisted driving, intelligent robots, and intelligent video surveillance. This research also contributes to education through curriculum development, student training, and knowledge dissemination. It includes interactions with K-12 students for participation and research opportunities. This research is to develop a unified visual compositional model that can effectively learn complex semantic concepts in a scalable end-to-end fashion, while achieving good generalizability and providing explainable parsing of the visual data. The project is focused on: (1) a principled model and its theoretical foundation, by designing a stochastic grammar based on the probabilistic And/Or-Graph to model the structural composition; (2) an effective computational approach for learning and parsing, by exploiting data-driven pattern mining to discover structural components and by exploring how the patterns may be self-formed; (3) a solid case study on video human activity parsing and interpretation, by inferring the complex compositions of human actions, body movements, and interaction with the environment; and (4) tools and prototype systems for human articulated body pose estimation, contextual object discovery, and video-based human activity analysis and interpretation.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.