RI: Small: Coordinating Multi-Agent Learning through Emergent Distributed Supervisory Control

该项目的重点是通过使用多智能体强化学习（MARL），为在不确定环境中运行的大规模多智能体系统开发协调策略。现有的MARL技术不能很好地扩展。本研究通过使用协调技术来“协调”个体智能体学习，从而加速收敛并导致学习策略更好地反映整体系统目标，从而解决了尺度问题。这个新颖的想法是通过一个低开销的紧急监督组织来实现的，该组织利用非局部信息来动态协调和塑造个体代理的学习过程，同时仍然允许代理对局部反馈自主反应。一个关键问题是如何使监督控制过程（包括监督信息的生成和组织的形成）的开发自动化。自动化的一种方法是使用代理之间的正式交互模型，该模型还包括代理的全局系统目标和策略空间模型，以获得适当监督控制所需的信息。另一种方法是将监督问题表述为分布式约束优化问题。这项工作的结果为各种下一代自适应应用的发展提供了必要的组成部分，如智能电网、云计算和大规模传感器网络。更广泛的影响源于所产生的学习技术在分布式控制、马萨诸塞大学本科生和研究生教育活动中的广泛适用性、使实验领域和算法公开可用的传播努力以及国际合作的发展。

The project is focused on developing coordination policies for large-scale multi-agent systems operating in uncertain environments through the use of multi-agent reinforcement learning (MARL). Existing MARL techniques do not scale well. This research addresses the scaling issue by using coordination technology to "coordinate" the individual agent learning so as to speed up convergence and lead to learned policies that better reflect overall system objectives. This novel idea is being implemented using an emergent supervisory organization with low overhead that exploits non-local information to dynamically coordinate and shape the learning processes of individual agents while still allowing agents to react autonomously to local feedback. A key question is how to automate the development of the supervisory control process (including supervisory information generation and organization formation). One approach to automation is using a formal model of interactions among agents that also includes a model of global system objectives and policy space of agents to derive the information necessary for appropriate supervisory control. Another approach is the formulation of the supervision problem as a distributed constraint optimization problem. The results of this work provide a necessary component for the development of a wide variety of next-generation adaptive applications, such as smart power grids, cloud computing, and large-scale sensor networks. The broader impact stems from the wide applicability of the resulting learning technology for distributed control, undergraduate and graduate educational activities at UMass, dissemination efforts that make the experimental domain and algorithms publically available, and the development of international collaborations.