MULTI-AGENT REINFORCEMENT LEARNING WITH NEUROEVOLUTION

已经报道了几种尝试，让多个单体强化学习（RL）代理合成高度协调的行为，以有效地实现其共同目标。大多数RL的直接应用都不能很好地扩展到更复杂的多智能体学习问题，因为每个RL智能体的状态空间随着参与联合任务的伙伴智能体的数量呈指数级增长。为了科普多智能体RL（MARL）中呈指数级增长的状态空间，我们以前提出了一种MARL方案，基于智能体决策策略的神经网络表示及其用实数编码遗传算法的优化，并通过对这些多个项目的应用表明了该方案的有效性，使用任何其它常规MARL方案都不能适当解决的代理学习问题，例如异步多代理跷跷板平衡问题和蜂窝电话系统中的动态信道分配问题。然而，由于该方案需要大量的计算资源，不能直接应用于大规模多智能体系统的设计问题。为了弥补这一缺陷，我们提出了一个大规模的MAS分层设计方案，它简单地分解成它的子任务的MAS的整个任务层次和优化的每个子任务与上述MARL计划。该设计方案的有效性，通过其应用程序的RoboCup足球队的设计问题，球队的任务被分解为原始的行动，由足球代理，行动之间的相互作用和协调的代理。

Several attempts have been reported to let multiple monolithic reinforcement learning (RL) agents synthesize highly coordinated behavior needed to accomplish their common goal effectively. Most of these straightforward application of RL scale poorly to more complex multi-agent learning problems, because the state space for each RL agent grows exponentially with the number of its partner agents engaged in the joint task.To cope with the exponentially large state space in multi-agent RL (MARL), we previously proposed a MARL scheme, based on neural network representation of the decision policy for an agent and its optimization with a real-coded GA, and showed the effectiveness of the scheme through its application to those multi-agent learning problems that can not be solved appropriately using any other conventional MARL scheme, such as the asynchronous multi-agent seesaw balancing problem and the dynamic channel allocation problem in cellular telephone systems.However, we can not apply the scheme directly to the design problems of large-scale multi-agent systems (MASs), because the scheme needs a huge amount of computation resources. To remedy the drawback, we propose a hierarchical design scheme of a large-scale MAS, which simply decomposes the whole task of the MAS into its subtasks hierarchically and optimizes each of the subtasks with the above-mentioned MARL scheme. The effectiveness of the design scheme is shown through its application to the RoboCup soccer team design problem where the task of the team is decomposed into the primitive actions by the soccer agents, interaction among the actions and coordination by the agents.