III: Small: Distributed Reinforcement Learning over Complex Networks

在许多分布式系统中，一组自治的代理必须在复杂的环境中协作，处理大量的流数据，同时做出最优决策。传统的决策方法很难适应这种情况，强化学习(RL)是一种很有前途的大规模分布式系统决策方法。然而，以前的分布式RL模型无法考虑代理的感知和观察能力，因此依赖于全局信息，而这在分布式环境中是不容易获得的。为了填补这一空白，该项目旨在为大规模联网系统构建一个革命性的、完全分布式的RL系统，而不使用全球信息。为此，该项目开发了设计、分析和测试完全分布式RL算法所需的新理论框架、计算模型和科学软件工具。这些算法将进一步设计成对动态环境具有健壮性，对对手攻击具有弹性，这将使多个自治代理的团队能够可靠地实现他们的目标。这项研究将对需要分布式机器学习算法和决策方法的实际应用领域产生重大影响。典型的例子包括移动机器人团队的运动规划，以及物联网环境中联网智能设备的协调。该项目促进教育和外联活动，包括扩大女学生在机器学习领域的参与，创建新课程，并为K-12学生和本科生设计研究项目。这些出版物和软件工具将与社区共享，以促进对分布式RR的进一步研究。该项目的中心目标是为大规模网络上的完全分布式RL算法的设计、分析和应用奠定理论基础。关键的技术挑战包括弥合全球可观测性设置和本地可观测性设置之间的差距，以及在动态和不可信任的通信存在的情况下实现弹性。为了实现技术目标和应对技术挑战，该项目调查了三个主要推力。第一个推力通过分布式估计近似全局信息，为全分布式RL的设计奠定了新的理论基础。第二个推力是针对时变的通信和感知能力、通信延迟和异步更新开发健壮的分布式RL算法。第三个推力设计了分布式RL算法，该算法首先设计了通信高效的RL算法，其中每个代理只能传输低维状态，然后设计了适用于小维甚至单维情况的弹性信息融合/聚合方法，从而对能够将不可信信息引入通信网络的对手和恶意攻击具有弹性。该项目提供了一套新颖的分布式RL算法，可以用于任何需要完全分布式决策和对流数据学习的应用领域，以及在对抗性环境中。同时，该项目还设计、开发和维护了一个软件框架，用于对整个分布式RL社区可以使用的分布式RL算法进行经验验证和研究。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

In many distributed systems, a team of autonomous agents must collaborate in a complex environment, process massive amounts of streaming data, and simultaneously make optimal decisions. Traditional decision-making techniques can hardly tackle such a scenario, and reinforcement learning (RL) has been recently shown to be a promising decision-making technique for large-scale distributed systems. However, previous distributed RL models have failed to account for sensing and observing capabilities of agents, and thus rely on global information, which is not readily available in distributed environments. To fill this gap, this project aims to build a revolutionary, fully distributed RL system for large-scale networked systems without using global information. Toward this end, the project develops a novel theoretical framework, computational models, and scientific software tools needed to design, analyze, and test fully distributed RL algorithms. The algorithms will be further designed to be robust against dynamic environments and resilient to adversarial attacks, which will enable teams of multiple autonomous agents to reliably achieve their goals. The research will greatly impact real-world application areas where distributed machine learning algorithms and decision-making methods are needed. Typical examples include motion planning of teams of mobile robots, and coordination of networked smart devices in an IoT environment. The project promotes education and outreach activities, including broadening participation of female students in the field of machine learning, creating new courses, and designing research projects for K-12 students and undergraduates. The publications and software tools will be shared with the community to foster further research on distributed RL.The central goal of this project is to establish theoretical foundations for fully distributed RL algorithm design, analysis, and applications over large-scale networks. The key technical challenges include bridging the gap between the global and local observability settings and achieving resiliency in the presence of dynamic and untrustworthy communications. To achieve the technical objective and tackle technical challenges, the project investigates three main thrusts. The first thrust establishes the fundamental novel theory for the design of fully distributed RL by approximating global information via distributed estimation. The second thrust develops robust distributed RL algorithms against time-varying communication and sensing capabilities, communication delays, and asynchronous updating. The third thrust designs distributed RL algorithms that are resilient to adversaries and malicious attacks capable of introducing untrustworthy information into the communication network, by first designing communication-efficient RL algorithms in which each agent can transmit only low-dimensional states, and then designing resilient information fusion/aggregation approaches for small- and even single-dimensional cases. The project provides a suite of novel distributed RL algorithms which can be used in any applied area where fully distributed decision making and learning with streaming data and in adversarial environments are needed. Concurrently with the three main thrusts, the project also designs, develops, and maintains a software framework for empirically validating and studying distributed RL algorithms that the entire distributed RL community can use.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.