A CO-EVOLUTIONARY MULTI-AGENT REINFORCEMENT LEARNING SCHEME TAKING ACCOUNT OF APPLICATION TO COMPETITIVE ENVIRONMENTS

已经报道了几种尝试，让多个单体强化学习（RL）代理合成高度协调的行为，以有效地实现其共同目标。大多数RL的直接应用都不能很好地扩展到更复杂的多智能体学习问题，因为每个RL智能体的状态空间随着参与联合任务的伙伴智能体的数量呈指数级增长。为了科普多智能体RL（MARL）中呈指数级增长的状态空间，我们以前提出了一种MARL方案，基于智能体决策策略的神经网络表示及其用实数编码遗传算法的优化，并通过对这些多个项目的应用表明了该方案的有效性，代理学习问题不能使用任何其他传统的MARL框架适当地解决。然而，一般来说，MARL方案在竞争环境中不起作用，因为我们不能预先提供任何绝对的个体适应度函数，因此我们不能应用实数编码的遗传算法。在竞争环境中，如一对一的比赛，个体的适应度是通过与其他个体的竞争，而不是通过一个绝对的适应度测量.为了弥补这一缺陷，我们扩展了MARL计划，取代其世代交替模型，MGG，由共同进化模型，称为CMGG，并允许群体中的个体（多智能体系统）通过相互竞争来共同进化地改进他们的策略。扩展MARL方案的有效性，通过其应用到一对一的函数逼近比赛和各种版本的二维空气曲棍球游戏。实验结果表明，该扩展方案的性能优于Floreano等人提出的最佳协同进化世代交替方案。

Several attempts have been reported to let multiple monolithic reinforcement learning (RL) agents synthesize highly coordinated behavior needed to accomplish their common goal effectively. Most of these straightforward application of RL scale poorly to more complex multi-agent learning problems, because the state space for each RL agent grows exponentially with the number of its partner agents engaged in the joint task.To cope with the exponentially large state space in multi-agent RL (MARL), we previously proposed a MARL scheme, based on neural network representation of the decision policy for an agent and its optimization with a real-coded GA, and showed the effectiveness of the scheme through its application to those multi-agent learning problems that can not be solved appropriately using any other conventional MARL frameworks.In general, however, the MARL scheme does not function in competitive environments, because we can not provide any absolute individual fitness functions in advance and accordingly we can not apply the real-coded GA. In competitive environments, such as one-on-one contests, individual fitness is evaluated through competition with other individuals, rather than through an absolute fitness measure.To remedy the drawback, we extend the MARL scheme by replacing its generation alternation model, MGG, by the co-evolutionary model, called CMGG, and allow individuals (multi-agent systems) in the population to co-evolutionarily improve their policies through competition with each other. The effectiveness of the extended MARL scheme is shown through its application to the one-on-one function approximation contest and various versions of the two-dimensional air-hockey games. The experimental results show that the extended scheme performs better over one of the best co-evolutionary generation alternation scheme proposed by Floreano and his colleagues.