A CO-EVOLUTIONARY MULTI-AGENT REINFORCEMENT LEARNING SCHEME TAKING ACCOUNT OF APPLICATION TO COMPETITIVE ENVIRONMENTS

据报道，有几种尝试让多个单片强化学习（RL）智能体合成高度协调的行为，以有效地完成它们的共同目标。大多数直接的强化学习应用都不适用于更复杂的多智能体学习问题，因为每个强化学习智能体的状态空间随着参与联合任务的伙伴智能体的数量呈指数增长。针对多智能体强化学习（MARL）中指数级大的状态空间，提出了一种基于智能体决策策略的神经网络表示及其实数编码遗传算法优化的MARL方案，并将该方案应用于其他传统MARL框架无法解决的多智能体学习问题，证明了该方案的有效性。然而，一般来说，MARL方案不能在竞争环境中发挥作用，因为我们不能事先提供任何绝对的个体适应度函数，因此我们不能应用实编码遗传算法。在竞争环境中，比如一对一的比赛，个体的适应性是通过与其他个体的竞争来评估的，而不是通过绝对的适应性测量。为了弥补这一缺陷，我们扩展了MARL方案，将其代交替模型MGG替换为共同进化模型CMGG，并允许种群中的个体（多智能体系统）通过相互竞争来共同进化地改进它们的策略。通过对一对一函数逼近竞赛和各种版本的二维空气曲棍球比赛的应用，证明了扩展MARL方案的有效性。实验结果表明，该扩展方案优于Floreano等提出的最佳协同进化代交替方案之一。

Several attempts have been reported to let multiple monolithic reinforcement learning (RL) agents synthesize highly coordinated behavior needed to accomplish their common goal effectively. Most of these straightforward application of RL scale poorly to more complex multi-agent learning problems, because the state space for each RL agent grows exponentially with the number of its partner agents engaged in the joint task.To cope with the exponentially large state space in multi-agent RL (MARL), we previously proposed a MARL scheme, based on neural network representation of the decision policy for an agent and its optimization with a real-coded GA, and showed the effectiveness of the scheme through its application to those multi-agent learning problems that can not be solved appropriately using any other conventional MARL frameworks.In general, however, the MARL scheme does not function in competitive environments, because we can not provide any absolute individual fitness functions in advance and accordingly we can not apply the real-coded GA. In competitive environments, such as one-on-one contests, individual fitness is evaluated through competition with other individuals, rather than through an absolute fitness measure.To remedy the drawback, we extend the MARL scheme by replacing its generation alternation model, MGG, by the co-evolutionary model, called CMGG, and allow individuals (multi-agent systems) in the population to co-evolutionarily improve their policies through competition with each other. The effectiveness of the extended MARL scheme is shown through its application to the one-on-one function approximation contest and various versions of the two-dimensional air-hockey games. The experimental results show that the extended scheme performs better over one of the best co-evolutionary generation alternation scheme proposed by Floreano and his colleagues.