Collaborative Research: AF: Small: Parallel Reinforcement Learning with Communication and Adaptivity Constraints

强化学习近年来取得了很大的研究进展，并在许多实际应用中取得了成功。然而，强化学习算法也有大规模应用需要大量数据和计算的名声。本项目将通过研究如何通过引入多个学习代理并允许它们收集数据并协作学习最优策略来使强化学习算法具有可扩展性这一重要问题来解决这一问题。该项目的结果将对大规模使用强化学习的许多领域产生影响，例如多阶段临床试验、培训自动驾驶算法、众包任务、定价和不同地点商店的分类优化。这些研究成果将通过在学术会议和研讨会、大学、工业实验室和在线媒体上的演讲进行传播，还将被整合到关于强化学习和大数据算法前沿的两门课程中。更严格地说，这个项目将研究如何解决学习代理在沟通和适应性方面的基本限制。具体地说，本项目将研究几种协作学习模型，包括完全通信、同步通信、有限自适应的同步通信和异步通信，并研究以下一般性问题：(1)并行学习模型中允许自适应的基本优势是什么；(2)基于模型的和非模型的强化学习在并行度上是否存在内在差异；(3)异步通信的影响是什么；以及(4)在强化学习中是否可以通信高效地并行化通用算法技术？研究团队将通过研究一系列核心问题来解决这些问题，包括多臂匪徒中的最佳ARM(S)识别和后悔最小化、上下文匪徒、有限状态马尔可夫决策过程学习、带函数逼近的强化学习以及MDP中的协调探索。通过对这些问题的研究，本项目将为通信高效的并行强化学习带来新的技术、视角和见解。该项目还将对控制理论、运筹学、信息理论和通信复杂性以及多智能体系统等相关研究领域产生重大影响。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Reinforcement learning has witnessed great research advancement in recent years and achieved successes in many practical applications. However, reinforcement-learning algorithms also have the reputation for being data- and computation-hungry for large-scale applications. This project will address this issue by studying the important question of how to make reinforcement-learning algorithms scalable via introducing multiple learning agents and allowing them to collect data and learn optimal strategies collaboratively. The outcomes of this project will have impacts on numerous areas where reinforcement learning is used at a scale, e.g., multi-phase clinical trials, training autonomous-driving algorithms, crowdsourcing tasks, pricing, and assortment optimization for stores at different locations. The research products will be disseminated via talks at academic conferences and workshops, universities, industrial labs, and online media, and will also be integrated in two courses on the forefront of reinforcement learning and big-data algorithms.More technically, this project will study how to address the fundamental constraints on communication and adaptivity for the learning agents. In particular, this project will investigate a handful of collaborative learning models, including full communication, synchronized communication, synchronized communication with limited adaptivity, and asynchronized communication, and study the following general questions: (1) what is the fundamental advantage of allowing adaptivity in the parallel learning model; (2) are there inherent differences on the degree of parallelism between model-based and model-free reinforcement learning; (3) what is the impact of asynchronized communication; and (4) is it possible to communication-efficiently parallelize general algorithmic techniques in reinforcement learning? The team of researchers will address these questions by studying a set of core problems, including best arm(s) identification and regret minimization in multi-armed bandits, contextual bandits, finite-state Markov decision process (MDP) learning, reinforcement learning with function approximates, and coordinated exploration in MDPs. Through studying these questions, this project will bring new techniques, perspectives, and insight to communication-efficient parallel reinforcement learning. This project will also have a significant impact on a number of related research areas such as control theory, operations research, information theory and communication complexity, and multi-agent systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.