CAREER:Towards Causal Multi-Modal Understanding with Event Partonomy and Active Perception

事件是复杂动态环境中因果关系和视觉理解的核心。从协助人类完成复杂任务的协作机器人到检测异常行为的监控系统，都需要了解事件、其组成及其相互作用，以实现有效的机器感知。该项目将探讨事件如何在多模态数据中结构化，以及如何利用它们来帮助设计更好的，具体的代理，可以构建和利用组合事件表示来帮助在复杂的现实环境中发挥作用。所开发的算法可能会在包括人工智能（AI）和教育在内的众多领域产生广泛影响，例如劳动力培训的未来。除了科学影响外，该项目还开展补充性的教育和外联活动。具体而言，它使更广泛的科学界参与人工智能和计算机视觉（CV）研究的使用，以通过研讨会和研讨会来增加劳动力培训的未来，介绍和加强俄克拉荷马州州立大学的人工智能和CV教育，并通过综合教育活动，在计算机科学教育和研究中发展和培养创业精神。研究重点是基于能量的神经网络的想法，符号学习，使用Grenander的模式理论形式主义，溯因推理和积极的具体视觉学习，并使用时间因果关系进行更丰富的多模态事件理解。该项目的具体研究目标有三个方面。首先，它试图通过以贝叶斯玫瑰树的形式表达层次结构来学习常见的日常事件的partonomy，贝叶斯玫瑰树的语义由基于能量的模式理论推理引擎填充。其次，它将研究如何利用此事件partonomy来理解视频中无法识别的动作，并在正在执行的整体任务的背景下感知当前动作。这种推理机制将使一个具体化的智能代理能够识别当前的动作，并在一个统一的基于能量的框架中推断出更高层次的概念，如人类的意图和目标。第三，它将实现partonomy为基础的理解框架中体现的代理，同时增强它与积极的多模态反馈。它将允许具体代理通过控制其几何参数（如位置，方向和姿势）来导航混乱并解决感知事件结构中的任何模糊性，从而通过来自环境的反馈执行主动推理。该项目由强大的情报（RI）计划和刺激竞争力研究的既定计划（EPSCoR）共同资助。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Events are central to causal, visual understanding in complex, dynamic environments. From collaborative robots that assist humans with complex tasks to surveillance systems that detect anomalous behavior, there is a need to understand events, their composition, and their interaction for effective machine perception. This project will explore how events are structured in multimodal data and how they can be leveraged to help design better, embodied agents that can construct and leverage compositional event representations to help function in complex, real-world environments. The developed algorithms could have a broad impact in numerous fields including Artificial Intelligence (AI) and education, such as the future of workforce training. In addition to scientific impact, the project performs complementary educational and outreach activities. Specifically, it engages the broader scientific community in the use of AI and computer vision (CV) research to augment the future of workforce training through workshops and seminars, introduces and enhances the AI and CV education at Oklahoma State University, and develops and fosters an entrepreneurial mindset in computer science education and research through integrated educational activities.The research focuses on the ideas of energy-based neuro-symbolic learning, using Grenander’s Pattern Theory formalism, abductive reasoning, and active embodied vision for learning and using temporal causality for richer, multimodal event understanding. The specific research aims of the project are three-fold. First, it seeks to learn the partonomy of common, everyday events by expressing the hierarchical structure in the form of Bayesian Rose Trees, whose semantics are populated by an energy-based pattern theory inference engine. Second, it will research ways to leverage this event partonomy into understanding actions in videos beyond recognition and perceive the current action in the context of the overall task being performed. This inference mechanism will enable an embodied, intelligent agent to recognize the current action and infer higher-level concepts such as human intent and goals in a unified energy-based framework. Third, it will realize the partonomy-based understanding framework in an embodied agent while augmenting it with active multimodal feedback. It will allow the embodied agent to perform active reasoning through feedback from the environment by controlling its geometric parameters (such as position, orientation, and pose) to navigate clutter and resolve any ambiguity in the perceived event structure. This project is jointly funded by Robust Intelligence (RI) Program and the Established Program to Stimulate Competitive Research (EPSCoR).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.