CAREER:Towards Causal Multi-Modal Understanding with Event Partonomy and Active Perception

在复杂、动态的环境中，事件是因果关系和视觉理解的核心。从协助人类完成复杂任务的协作机器人到检测异常行为的监视系统，都需要了解事件、它们的组成以及它们之间的相互作用，以实现有效的机器感知。该项目将探索如何在多模态数据中构建事件，以及如何利用它们来帮助设计更好的嵌入代理，这些代理可以构建和利用组合事件表示来帮助在复杂的现实世界环境中发挥作用。开发的算法可能会对包括人工智能（AI）和教育在内的许多领域产生广泛影响，例如未来的劳动力培训。除了科学影响外，该项目还开展了补充的教育和推广活动。具体来说，它让更广泛的科学界参与到人工智能和计算机视觉（CV）研究的使用中，通过研讨会和研讨会来增加劳动力培训的未来，介绍并加强俄克拉荷马州立大学的人工智能和CV教育，并通过综合教育活动发展和培养计算机科学教育和研究中的企业家心态。研究的重点是基于能量的神经符号学习思想，利用格林纳德模式理论的形式主义、溯因推理和主动具身视觉进行学习，并利用时间因果关系进行更丰富的多模态事件理解。该项目的具体研究目的有三个方面。首先，它试图通过以贝叶斯玫瑰树的形式表达层次结构来学习常见的日常事件的部分分类法，其语义由基于能量的模式理论推理引擎填充。其次，它将研究如何利用这一事件分类学来理解视频中无法识别的动作，并在正在执行的整体任务的背景下感知当前动作。这种推理机制将使具体化的智能代理能够识别当前的动作，并在统一的基于能量的框架中推断出更高层次的概念，如人类意图和目标。第三，它将在具身代理中实现基于局部经济学的理解框架，同时通过主动多模态反馈对其进行增强。它将允许嵌入的智能体通过控制其几何参数（如位置、方向和姿态）从环境反馈中执行主动推理，以导航混乱并解决感知到的事件结构中的任何模糊性。该项目由健全情报（RI）计划和促进竞争研究的既定计划（EPSCoR）联合资助。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Events are central to causal, visual understanding in complex, dynamic environments. From collaborative robots that assist humans with complex tasks to surveillance systems that detect anomalous behavior, there is a need to understand events, their composition, and their interaction for effective machine perception. This project will explore how events are structured in multimodal data and how they can be leveraged to help design better, embodied agents that can construct and leverage compositional event representations to help function in complex, real-world environments. The developed algorithms could have a broad impact in numerous fields including Artificial Intelligence (AI) and education, such as the future of workforce training. In addition to scientific impact, the project performs complementary educational and outreach activities. Specifically, it engages the broader scientific community in the use of AI and computer vision (CV) research to augment the future of workforce training through workshops and seminars, introduces and enhances the AI and CV education at Oklahoma State University, and develops and fosters an entrepreneurial mindset in computer science education and research through integrated educational activities.The research focuses on the ideas of energy-based neuro-symbolic learning, using Grenander’s Pattern Theory formalism, abductive reasoning, and active embodied vision for learning and using temporal causality for richer, multimodal event understanding. The specific research aims of the project are three-fold. First, it seeks to learn the partonomy of common, everyday events by expressing the hierarchical structure in the form of Bayesian Rose Trees, whose semantics are populated by an energy-based pattern theory inference engine. Second, it will research ways to leverage this event partonomy into understanding actions in videos beyond recognition and perceive the current action in the context of the overall task being performed. This inference mechanism will enable an embodied, intelligent agent to recognize the current action and infer higher-level concepts such as human intent and goals in a unified energy-based framework. Third, it will realize the partonomy-based understanding framework in an embodied agent while augmenting it with active multimodal feedback. It will allow the embodied agent to perform active reasoning through feedback from the environment by controlling its geometric parameters (such as position, orientation, and pose) to navigate clutter and resolve any ambiguity in the perceived event structure. This project is jointly funded by Robust Intelligence (RI) Program and the Established Program to Stimulate Competitive Research (EPSCoR).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.