A CO-EVOLUTIONARY MULTI-AGENT REINFORCEMENT LEARNING SCHEME TAKING ACCOUNT OF APPLICATION TO COMPETITIVE ENVIRONMENTS

点击翻译按钮获取中文摘要

Several attempts have been reported to let multiple monolithic reinforcement learning (RL) agents synthesize highly coordinated behavior needed to accomplish their common goal effectively. Most of these straightforward application of RL scale poorly to more complex multi-agent learning problems, because the state space for each RL agent grows exponentially with the number of its partner agents engaged in the joint task.To cope with the exponentially large state space in multi-agent RL (MARL), we previously proposed a MARL scheme, based on neural network representation of the decision policy for an agent and its optimization with a real-coded GA, and showed the effectiveness of the scheme through its application to those multi-agent learning problems that can not be solved appropriately using any other conventional MARL frameworks.In general, however, the MARL scheme does not function in competitive environments, because we can not provide any absolute individual fitness functions in advance and accordingly we can not apply the real-coded GA. In competitive environments, such as one-on-one contests, individual fitness is evaluated through competition with other individuals, rather than through an absolute fitness measure.To remedy the drawback, we extend the MARL scheme by replacing its generation alternation model, MGG, by the co-evolutionary model, called CMGG, and allow individuals (multi-agent systems) in the population to co-evolutionarily improve their policies through competition with each other. The effectiveness of the extended MARL scheme is shown through its application to the one-on-one function approximation contest and various versions of the two-dimensional air-hockey games. The experimental results show that the extended scheme performs better over one of the best co-evolutionary generation alternation scheme proposed by Floreano and his colleagues.