CAREER: Principled Deep Reinforcement Learning for Societal Systems

深度强化学习（RL）最近的突破，特别是它在棋盘和电子游戏（如围棋、雅达利、Dota和星际争霸）中的超人水平的表现，为通过学习控制许多复杂和未知的系统开辟了新的途径。然而，在游戏之外的实际用途上，深度强化学习仍然缺乏效率和可信度。就效率而言，深度强化学习的经验成功需要数百万到数十亿个数据点和数天到数周的运行时间。就可信度而言，深度强化学习的经验成功仅由收到的奖励来衡量，而没有考虑安全性和鲁棒性。当我们将深度强化学习扩展到设计和优化关键领域的社会系统（例如医疗保健、交通、电网、金融网络和供应链）时，这种效率和可信度的缺乏会进一步加剧。本CAREER提案通过建立分析单智能体深度强化学习的计算效率和样本效率的理论框架以及实现这些效率的算法框架来解决这些挑战。此外，它还导致了一个随机博弈框架，用于通过多智能体深度强化学习在社会系统中实现安全性、鲁棒性、可扩展性、公平性、风险意识和激励。研究计划强调深度强化学习与多个领域的结合，如非凸优化、非参数统计、因果推理、随机博弈、社会科学等。该教育计划强调，在社会背景下，将数据驱动的决策作为一项基本技能传授给后代，尤其是未来的领导者。特别是，它旨在促进数据驱动型社会领导的理念，并支持未被充分代表的少数民族研究人员和学生，他们亲身经历了从K-12教育到研究生培训等社会系统中的紧迫挑战。为了应对持续的大流行，外展计划包括组织关于数据科学和人工智能的在线研讨会，通过整合研究和教育来指导远程实习生，并通过数据测试和客户端项目挑战吸引远程学生。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The recent breakthrough in deep reinforcement learning (RL), especially its superhuman-level performance in board and video games, e.g., Go, Atari, Dota, and StarCraft, opens up new avenues for controlling many complex and unknown systems via learning. However, for practical purposes beyond game playing, deep RL still suffers from a lack of efficiency and trustworthiness. In terms of efficiency, the empirical success of deep RL requires millions to billions of data points and days to weeks of running time. In terms of trustworthiness, the empirical success of deep RL is only measured by the received reward, which does not account for safety and robustness. Such a lack of efficiency and trustworthiness is further exacerbated when we scale up deep RL to design and optimize societal systems in critical domains, e.g., healthcare, transportation, power grid, financial network, and supply chain.This CAREER proposal addresses these challenges by establishing a theoretical framework for analyzing the computational efficiency and sample efficiency of single-agent deep RL and an algorithmic framework for achieving such efficiencies. Moreover, it leads to a stochastic game framework for achieving safety, robustness, scalability, fairness, risk-awareness, and incentivization in social systems via multi-agent deep RL. The research plan emphasizes connecting deep RL with multiple fields, e.g., nonconvex optimization, nonparametric statistics, causal inference, stochastic game, and social science. The education plan emphasizes teaching data-driven decision making as a fundamental skill for future generations, especially for future leaders, in societal contexts. In particular, it aims to promote the idea of data-driven social leadership and support underrepresented minority researchers and students, who personally experience pressing challenges in societal systems, from K-12 education to graduate training. In order to cope with the ongoing pandemic, the outreach plan involves organizing online seminars on data science and artificial intelligence, mentoring remote interns by integrating research and education, and engaging remote students via DataFest and Client Project Challenge.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.