Cooperative Coevolution of Neural Networks in Sequential Decision Tasks

在诸如资源优化、机器人控制和游戏等顺序决策任务中，在评估结果之前必须做出几个决策。这种强化反馈依赖于决策的整个序列，很难确定哪个决策对结果负责。该项目旨在开发更好的技术，以遗传算法进化的神经网络为基础，在具有这种稀疏反馈的领域进行学习。目标是既能够更快地解决现有问题，又能够解决以前作为顺序决策任务不可行的问题。我们之前的工作表明，当单个神经元进化成合作并形成良好的网络时，神经进化是最强大的。本课题对这种协同进化方法进行了深入的研究。这项研究旨在回答三个主要问题：合作共同进化的力量从何而来？利用这种力量的最佳方式是什么？进化强化学习方法在学习顺序决策任务方面与传统的价值函数方法有何不同？进化强化学习在现实应用中是否具有所需的准确性和灵活性？如果成功，该项目将产生协同进化算法，该算法将更快地解决现有的顺序决策任务，并允许解决比以前更困难的任务。我们将了解如何在给定的强化学习任务中选择进化方法和值函数方法，以及如何最有效地使用每种方法。最后，该项目将展示一般的学习，特别是神经网络的合作协同进化，如何用于节省资源，并在具有挑战性的现实世界任务中实现复杂的行为。

In sequential decision tasks such as resource optimization, robot control, and game playing, several decisions must be made before the outcome can be evaluated. Such reinforcement feedback depends on the entire sequence of decisions, and it is difficult to determine which of the decisions were responsible for the outcome. This project aims at developing better techniques for learning in domains with such sparse feedback, based on evolving neural networks with genetic algorithms. The goal is both to be able to solve existing problems faster, and to be able to solve problems that have not been feasible as sequential decision tasks before. Our previous work showed that neuroevolution is most powerful when individual neurons are evolved to cooperate and form good networks. In this project, such cooperative coevolution methods are studied in depth. The research aims at answering three main questions: Where does the power of cooperative coevolution come from and what are the best ways of making use of it? How do the evolutionary reinforcement learning methods differ from the traditional value function methods in learning sequential decision tasks? Does evolutionary reinforcement learning have the accuracy and flexibility required in real-world applications? If successful, the project will result in cooperative coevolution algorithms that will solve existing sequential decision tasks faster, and will allow solving more difficult tasks than before. We will know how to decide between evolutionary and value function methods for a given reinforcement learning task, and also how to use each method most effectively. Finally, the project will demonstrate how learning in general, and cooperative coevolution of neural networks in particular, can be used to save resources and achieve complex behavior in challenging real-world tasks.