CAREER: Foundations of Reinforcement Learning under Partial Observability

广泛的现代人工智能挑战可以被视为部分可观察性下的强化学习（RL）问题，其中代理学习做出一系列决策，尽管缺乏关于做出决策的即时情况的完整信息。这种部分可观察RL （PORL）的自然应用包括机器人、自动驾驶、不完全信息博弈、部分信息下的资源分配、行星探索、医疗诊断系统等。因此，PORL已经成为运筹学、控制和机器学习中的一个重要课题。虽然社区最近见证了在完全可观察环境中强化学习理论的突破，但我们对学习在部分可观察系统中采取行动的理解仍然非常有限。部分可观测性给强化学习在建模、算法设计和理论分析等方面带来了一系列新的独特挑战。解决这些挑战将对现代RL可以应用的学术界，工业界和社会产生深远的影响。本项目旨在识别和解决这些独特的挑战，建立坚实的理论基础，并为PORL设计新的可靠和高效的算法。具体地说，这项建议将分三个阶段逐步研究PORL。推力1考虑在部分可观察马尔可夫决策过程（pomdp）模型下的基本表格设置。这一主旨的主要目标是确定允许统计或计算高效学习的关键结构条件，并解决推断潜在状态和探索的核心挑战。推力2涉及具有大量状态和观察的现代PORL，其中必须部署函数近似来近似模型、值函数或策略。我们将在更一般的预测状态表示（PSRs）模型下研究这些问题，并在函数近似的存在下开发有效的学习结果。在部分可观察马尔可夫博弈（POMGs）模型下，研究多智能体环境下的PORL。我们将设计有效的算法来学习pomg中的各种均衡，并解决多机构和分散算法设计所带来的独特挑战。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

A wide range of modern artificial intelligence challenges can be cast as Reinforcement Learning (RL) problems under partial observability, in which agents learn to make a sequence of decisions despite lacking complete information about the moment-to-moment situation in which decisions are made. Natural applications of this kind of Partially Observable RL (PORL) include robotics, autonomous driving, imperfect information games, resource allocation under partial information, planetary exploration, medical diagnostic systems. As such, PORL has been an important topic in operation research, control, and machine learning. While the community recently witnessed a surge of breakthroughs in reinforcement learning theory in fully observable environments, our understanding of learning to act in partially observable systems remains very limited. Partial observability brings a new series of unique challenges to RL in modeling, algorithm design, and theoretical analyses. Resolving these challenges will have far-reaching impacts in academia, industry and society where modern RL can be applied.This project aims to identify and attack these unique challenges, establish solid theoretical foundations, and design new reliable and efficient algorithms for PORL. Concretely, this proposal will study PORL in three progressive thrusts. Thrust 1 considers the basic tabular setup, under the model of Partially Observable Markov Decision Processes (POMDPs). The main objective in this thrust is to identify the key structural conditions that permit statistically or computationally efficient learning, and to address the core challenges of inferring latent states and exploration. Thrust 2 concerns modern PORL with an enormous number of states and observations, where function approximation must be deployed to approximate the models, the value functions, or the policies. We will investigate these problems under a more general model of Predictive State Representations (PSRs) and develop efficient learning results in the presence of function approximation. Thrust 3 investigates PORL in the multiagent setting, under the model of Partially Observable Markov Games (POMGs). We will design efficient algorithms for learning various equilibria in POMGs and address the unique challenges arising from multiagency and the design of decentralized algorithms.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.