CAREER: Temporal Causal Reinforcement Learning and Control for Autonomous and Swarm Cyber-Physical Systems

了解行为的根本原因对于知情决策和防止无效或有偏见的政策至关重要。目前，大多数嵌入在网络物理系统（CPS）中的基于人工智能的学习和控制模块依赖于统计相关性而不是因果关系来进行决策。这不仅会导致错误的决策，还会阻碍学习的可解释性，限制可转移性和可扩展性。该CAREER提案旨在弥合因果推理与CPS中不断增长的强化学习（RL）能力之间的差距。所提出的方法对广泛的CPS应用具有变革性，可以在自动驾驶汽车、无人机、工业机器人和群体机器人等自主和群体CPS中实现更高效和有效的决策过程。NSF CAREER提案通过在单智能体、多智能体和群体系统设置中利用时间逻辑和因果图的推理能力，为CPS提出了一套时间因果强化学习和控制方法。我们开发的工具将在多个CPS测试平台上实施，并与拟议的教育计划相结合。本文提出的算法具有以下独特和创新的特点：首先，我们将开发计算效率高的工具，可以从CPS执行RL的观测和干预数据中发现时间因果知识，以提高采样效率和可转移性。其次，我们将在合作、非合作和不完全信息随机博弈环境中开发用于CPS的多智能体强化学习方法，其中以分布式方式发现时间因果知识以加速强化学习。最后，我们将为群体系统开发可扩展的基于强化学习的控制方法，利用对代理级特征和群体级特征（如密度和广义矩）的时间因果推理。该教育计划将通过人工智能辅助的自适应和互动教学、基于时间逻辑的教育游戏、针对时间因果强化学习的在线互动教育网站设计以及与行业合作伙伴的研讨会和网络研讨会，影响下一代CPS和人工智能工程师和研究人员。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Understanding the root cause of behavior is imperative for informed decision-making and preventing ineffective or biased policies. Currently, most AI-based learning and control modules embedded in cyber-physical systems (CPS) rely on statistical correlation rather than causality for decision-making. This not only results in incorrect decisions but also hinders the interpretability of learning, limiting transferability and scalability. This CAREER proposal aims to bridge the gap between causal inference and the growing capabilities of reinforcement learning (RL) in CPS. The proposed methods are transformative to a wide range of CPS applications, enabling more efficient and effective decision-making processes in autonomous and swarm CPS such as self-driving cars, drones, industrial robots, and swarm robots.This NSF CAREER proposal proposes a set of temporal causal RL and control approaches for CPS by leveraging the reasoning capabilities of temporal logics and causal diagrams in single-agent, multi-agent, and swarm system settings. The tools we develop will be implemented on multiple CPS testbeds and integrated with the proposed education plan. The proposed algorithms have the following unique and innovative features. Firstly, we will develop computationally efficient tools that can discover temporal causal knowledge from both observational and interventional data of a CPS in performing RL to improve the sampling efficiency and transferability. Secondly, we will develop multi-agent RL approaches for CPS in cooperative, non-cooperative, and incomplete information stochastic game environments where temporal causal knowledge is discovered in a distributed way for expediting RL. Lastly, we will develop scalable RL-based control methods for swarm systems utilizing temporal causal reasoning over agent-level features and swarm-level features such as densities and generalized moments. The education plan will impact the next generation of CPS and AI engineers and researchers through AI-assisted adaptive and interactive teaching, temporal-logic-based educational games, online interactive educational website design for temporal causal RL, and workshops and webinars with industrial partners.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.