Improved learning of reactive, cooperative behaviour through learning-based testing

我研究的长期愿景是使用机器学习来改进分布式系统软件的开发。我的主要想法是修改和扩展合作行为学习的概念，以包括基于学习的测试组件的结果，该组件发现学习的合作行为的负面后果，包括预期的功能、效率和安全性方面的问题。*为了学习协作行为，我们将分布式系统的组件视为可以执行操作的代理。学习试图为这些代理创建控制，以便它们一起实现整个代理组所负责的功能。*使用基于学习的测试来发现负面结果是一种进化学习过程，其中测试组件创建与被测试系统的交互序列，并基于它们与测试目标的接近程度来评估序列。评估被用来学习定序器，这些定序器越来越接近于产生测试系统正在寻找的负面后果。*主要的研究挑战是如何将合作行为的学习重点与基于学习的测试的重点结合起来，试图处理系统可能处于的所有可能的情况，即找到揭示被测试系统的特定问题的特定情况序列。这一挑战的可能解决方案可以针对执行基于学习的测试的方式，概括其结果，但也可以使用例外规则修改一般学习者和体系结构。在早期的工作中，我们开发了面向代理的Shout-Ahead体系结构，并针对该体系结构提出了一种混合学习方法，将强化学习和进化学习相结合。为游戏《韦斯诺斯之战》开发敌方单位控制人工智能的概念验证系统表明，最好的人工智能能够击败游戏附带的相当好的人工智能。我们探索了使用行为学习来测试系统在几个应用程序(包括游戏)中的弱点。在所有这些应用中，我们的测试系统能够发现测试系统中的几个弱点，包括效率和安全弱点。*在这些工作的基础上，我们将在两个案例研究中探索我们的想法，将一般行为学习与基于学习的测试相结合，作为物联网应用的一个扩展系统和一个自动化牛群配送中心。这项研究的预期结果是，由于自动化，能够以比当前实践更低的成本和更高的质量创建定制的开放分布式系统。

The long-term vision for my research is to use machine learning to improve the development of distributed systems software. My key idea is to modify and extend a concept for learning of cooperative behaviour to include the results of a learning-based testing component that finds negative consequences of the learned cooperative behaviour, including problems with intended functionality, efficiency and security. ***To learn cooperative behaviour, we see the components of a distributed system as agents that can perform actions. The learning tries to create controls for these agents, so that they together achieve the functionality the whole group of agents is tasked with.***Using learning-based testing for finding negative consequences is an evolutionary learning process in which the testing component creates interaction sequences with the tested system and evaluates the sequences based on how near they come to a testing goal. The evaluation is used to learn sequencers that come nearer and nearer to creating the negative consequences the testing system is looking for.***The main research challenge is how to combine the focus of learning of cooperative behaviour on trying to deal with all possible situations the system can be in with the focus of learning-based testing on finding one particular sequence of situations revealing a particular problem of the tested system. Possible solutions of this challenge can target the way learning-based testing is performed, generalizing its results, but also modifying the general learner and the architecture with exception rules. In earlier work, we have developed the shout-ahead architecture for agents and a hybrid learning method for this architecture, combining reinforcement learning with evolutionary learning. A proof-of-concept system for developing control AIs for enemy units for the game Battle for Wesnoth showed that the best learned AIs are able to beat the rather good human-created central AI that comes with the game. We explored the usage of learning of behavior to test systems for weaknesses for several applications, including games. In all these applications, our test systems were able to find several weaknesses in the tested systems, including efficiency and security weaknesses.***Building on these works, we will explore our ideas for combining general learning of behaviour with learning-based testing in two case studies, an extension of the system for Battle for Wesnoth and an automated cattle distribution center as an example of Internet of Things applications. The anticipated result of this research is the ability to create customized open distributed systems at lower cost than current practise and with higher quality due to automization.