RI: Small: Coordinating Multi-Agent Learning through Emergent Distributed Supervisory Control

该项目致力于通过使用多智能体强化学习(MAIL)为运行在不确定环境中的大规模多智能体系统开发协调策略。现有的Marl技术不能很好地扩展。本研究通过使用协调技术来“协调”单个代理的学习来解决可伸缩性问题，以加快收敛并导致学习的策略更好地反映整体系统目标。这一新的想法是使用一个低开销的紧急监督组织来实现的，该组织利用非本地信息来动态协调和塑造单个代理的学习过程，同时仍然允许代理对本地反馈做出自主反应。一个关键问题是如何使监督控制过程(包括监督信息的产生和组织的形成)的发展自动化。实现自动化的一种方法是使用代理之间相互作用的正式模型，该模型还包括代理的全球系统目标和策略空间的模型，以获得适当监督控制所需的信息。另一种方法是将监督问题表示为分布式约束优化问题。这项工作的结果为智能电网、云计算和大规模传感器网络等各种下一代自适应应用的开发提供了必要的组成部分。更广泛的影响源于由此产生的学习技术在分布式控制、马萨诸塞州大学的本科生和研究生教育活动中的广泛适用性、使实验领域和算法公开的传播努力以及国际合作的发展。

The project is focused on developing coordination policies for large-scale multi-agent systems operating in uncertain environments through the use of multi-agent reinforcement learning (MARL). Existing MARL techniques do not scale well. This research addresses the scaling issue by using coordination technology to "coordinate" the individual agent learning so as to speed up convergence and lead to learned policies that better reflect overall system objectives. This novel idea is being implemented using an emergent supervisory organization with low overhead that exploits non-local information to dynamically coordinate and shape the learning processes of individual agents while still allowing agents to react autonomously to local feedback. A key question is how to automate the development of the supervisory control process (including supervisory information generation and organization formation). One approach to automation is using a formal model of interactions among agents that also includes a model of global system objectives and policy space of agents to derive the information necessary for appropriate supervisory control. Another approach is the formulation of the supervision problem as a distributed constraint optimization problem. The results of this work provide a necessary component for the development of a wide variety of next-generation adaptive applications, such as smart power grids, cloud computing, and large-scale sensor networks. The broader impact stems from the wide applicability of the resulting learning technology for distributed control, undergraduate and graduate educational activities at UMass, dissemination efforts that make the experimental domain and algorithms publically available, and the development of international collaborations.