MULTI-AGENT REINFORCEMENT LEARNING WITH NEUROEVOLUTION

据报道，有几种尝试让多个单片强化学习（RL）智能体合成高度协调的行为，以有效地完成它们的共同目标。大多数直接的强化学习应用都不适用于更复杂的多智能体学习问题，因为每个强化学习智能体的状态空间随着参与联合任务的伙伴智能体的数量呈指数增长。为了应对多智能体强化学习（MARL）中指数级大的状态空间，我们提出了一种基于智能体决策策略的神经网络表示及其实数编码遗传算法优化的MARL方案，并将该方案应用于其他传统MARL方案无法解决的多智能体学习问题，表明了该方案的有效性。例如蜂窝电话系统中的异步多智能体跷跷板平衡问题和动态信道分配问题。然而，由于该方案需要大量的计算资源，我们不能将其直接应用于大规模多智能体系统（MASs）的设计问题。为了弥补这一缺陷，我们提出了一种大规模MAS的分层设计方案，该方案将MAS的整个任务分层分解为子任务，并利用上述MARL方案对每个子任务进行优化。将该设计方案应用于机器人世界杯足球队设计问题，将足球队任务分解为足球agent的原始动作、agent之间的交互和agent之间的协调，证明了该设计方案的有效性。

Several attempts have been reported to let multiple monolithic reinforcement learning (RL) agents synthesize highly coordinated behavior needed to accomplish their common goal effectively. Most of these straightforward application of RL scale poorly to more complex multi-agent learning problems, because the state space for each RL agent grows exponentially with the number of its partner agents engaged in the joint task.To cope with the exponentially large state space in multi-agent RL (MARL), we previously proposed a MARL scheme, based on neural network representation of the decision policy for an agent and its optimization with a real-coded GA, and showed the effectiveness of the scheme through its application to those multi-agent learning problems that can not be solved appropriately using any other conventional MARL scheme, such as the asynchronous multi-agent seesaw balancing problem and the dynamic channel allocation problem in cellular telephone systems.However, we can not apply the scheme directly to the design problems of large-scale multi-agent systems (MASs), because the scheme needs a huge amount of computation resources. To remedy the drawback, we propose a hierarchical design scheme of a large-scale MAS, which simply decomposes the whole task of the MAS into its subtasks hierarchically and optimizes each of the subtasks with the above-mentioned MARL scheme. The effectiveness of the design scheme is shown through its application to the RoboCup soccer team design problem where the task of the team is decomposed into the primitive actions by the soccer agents, interaction among the actions and coordination by the agents.