RI: Small: Learning Strategic Behavior in Sequential Decision Tasks

许多日常的、现实世界的任务可以看作是顺序决策任务。例如，在复杂的环境中引导机器人，在拥挤的交通中驾驶汽车，以及在计算机网络中路由数据包，都需要做出一系列决策，从而最大限度地减少时间和资源的使用。将这些任务自动化是可取的，但这很困难，因为最佳决策通常是未知的。许多现有的学习方法会导致在短期内表现良好的反应性行为，但从长远来看并不能达到智能的高水平行为。这个项目正在开发学习战略高级行为的方法。策略方法需要(1)保留过去状态的信息；(2)学习多模态行为；(3)基于关键细节在不同行为之间进行选择；(4)基于这些行为实施顺序的高级策略。在先前的工作中开发的神经进化方法通过进化（通过遗传算法）递归神经网络来表示行为来解决第一个问题。为了解决剩下的问题，我们将这些方法扩展到多目标优化、级联结构的局部节点以及模块及其组合的演化。初步结果表明，这种方法确实可行。从长远来看，发达的技术将使为现实世界的任务构建健壮的顺序决策系统成为可能。它带来了更安全、更高效的车辆、交通和机器人控制，改进的过程和制造优化，以及更高效的计算机和通信系统。它还将使下一代电子游戏成为可能，其中的角色表现出现实的、战略性的行为：这种技术应该会在未来带来更有效的教育和训练游戏。在这项工作中开发的OpenNERO开源软件平台将提供给研究社区。

Many routine, real-world tasks can be seen as sequential decision tasks. For instance, navigating a robot through a complex environment, driving a car in congested traffic, and routing packets in a computer network requires making a sequence of decisions that together minimize time and resources used. It would be desirable to automate these tasks, yet it is difficult because the optimal decisions are generally not known. Many existing learning methods lead to reactive behaviors that perform well in short term, but do not amount to intelligent high-level behavior in the long term. This project is developing methods for learning strategic high-level behavior. Strategic methods need to (1) retain information from past states, (2) learn multimodal behavior, (3) choose between the different behaviors based on crucial detail, and (4) implement a sequential high-level strategy based on those behaviors. The neuroevolution methods developed in prior work solve the first problem by evolving (through genetic algorithms) recurrent neural networks to represent the behavior. To solve the remaining problems, these methods are being extended in the proposed work with multi-objective optimization, local nodes with cascaded structure, and with evolution of modules and their combinations. Preliminary results indicate that this approach is indeed feasible. In the long term, developed technology will make it possible to build robust sequential decision systems for real-world tasks. It leads to safer and more efficient vehicle, traffic, and robot control, improved process and manufacturing optimization, and more efficient computer and communication systems. It will also make the next generation of video games possible, with characters that exhibit realistic, strategic behaviors: Such technology should lead to more effective educational and training games in the future. The OpenNERO open source software platform developed in this work will be made available to the research community.