Collaborative Research: AF: Small: Parallel Reinforcement Learning with Communication and Adaptivity Constraints

近年来，强化学习的研究取得了很大的进展，并在许多实际应用中取得了成功。 然而，重复学习算法在大规模应用程序中也有数据和计算需求。 该项目将通过研究如何通过引入多个学习代理并允许它们协作收集数据和学习最佳策略来使重复学习算法可扩展的重要问题来解决这个问题。 该项目的成果将对强化学习大规模使用的许多领域产生影响，例如，多阶段临床试验、训练快速驾驶算法、众包任务、定价和不同地点商店的分类优化。 该研究成果将通过学术会议和研讨会、大学、工业实验室和在线媒体的讲座进行传播，并将整合到强化学习和大数据算法的两门前沿课程中。从技术上讲，该项目将研究如何解决学习代理在通信和自适应方面的基本限制。 特别是，本项目将研究几种协作学习模式，包括完全通信、同步通信、具有有限适应性的同步通信和并行通信，并研究以下一般性问题：（1）在并行学习模式中允许适应性的根本优势是什么;（2）基于模型和无模型的强化学习之间的并行度是否存在固有差异;（3）简化通信的影响是什么;以及（4）在强化学习中，是否可以通信有效地并行化一般算法技术？ 研究团队将通过研究一系列核心问题来解决这些问题，包括多臂土匪中的最佳手臂识别和遗憾最小化，上下文土匪，有限状态马尔可夫决策过程（MDP）学习，函数近似的强化学习以及MDP中的协调探索。 通过研究这些问题，该项目将为通信高效的并行强化学习带来新的技术，观点和见解。 该项目还将对控制理论、运筹学、信息论和通信复杂性以及多智能体系统等相关研究领域产生重大影响。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Reinforcement learning has witnessed great research advancement in recent years and achieved successes in many practical applications. However, reinforcement-learning algorithms also have the reputation for being data- and computation-hungry for large-scale applications. This project will address this issue by studying the important question of how to make reinforcement-learning algorithms scalable via introducing multiple learning agents and allowing them to collect data and learn optimal strategies collaboratively. The outcomes of this project will have impacts on numerous areas where reinforcement learning is used at a scale, e.g., multi-phase clinical trials, training autonomous-driving algorithms, crowdsourcing tasks, pricing, and assortment optimization for stores at different locations. The research products will be disseminated via talks at academic conferences and workshops, universities, industrial labs, and online media, and will also be integrated in two courses on the forefront of reinforcement learning and big-data algorithms.More technically, this project will study how to address the fundamental constraints on communication and adaptivity for the learning agents. In particular, this project will investigate a handful of collaborative learning models, including full communication, synchronized communication, synchronized communication with limited adaptivity, and asynchronized communication, and study the following general questions: (1) what is the fundamental advantage of allowing adaptivity in the parallel learning model; (2) are there inherent differences on the degree of parallelism between model-based and model-free reinforcement learning; (3) what is the impact of asynchronized communication; and (4) is it possible to communication-efficiently parallelize general algorithmic techniques in reinforcement learning? The team of researchers will address these questions by studying a set of core problems, including best arm(s) identification and regret minimization in multi-armed bandits, contextual bandits, finite-state Markov decision process (MDP) learning, reinforcement learning with function approximates, and coordinated exploration in MDPs. Through studying these questions, this project will bring new techniques, perspectives, and insight to communication-efficient parallel reinforcement learning. This project will also have a significant impact on a number of related research areas such as control theory, operations research, information theory and communication complexity, and multi-agent systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.