Improved learning of reactive, cooperative behaviour through learning-based testing

我研究的长期愿景是使用机器学习来改进分布式系统软件的开发。我的主要想法是修改和扩展合作行为学习的概念，以包括基于学习的测试组件的结果，该组件发现学习的合作行为的负面后果，包括预期功能，效率和安全性的问题。为了学习合作行为，我们将分布式系统的组件视为可以执行动作的代理。学习尝试为这些代理创建控制，以便它们一起实现整个代理组的任务功能。***使用基于学习的测试来发现负面结果是一个进化的学习过程，在这个过程中，测试组件创建与被测试系统的交互序列，并根据它们接近测试目标的程度来评估这些序列。评估用于学习越来越接近产生测试系统正在寻找的负面结果的测序器。***主要的研究挑战是如何将合作行为学习的重点与基于学习的测试的重点结合起来，学习的重点是试图处理系统可能遇到的所有可能的情况，而基于学习的测试的重点是找到一个特定的情况序列，揭示被测试系统的特定问题。这一挑战的可能解决方案可以针对基于学习的测试的执行方式，概括其结果，但也可以使用例外规则修改通用学习器和体系结构。在早期的工作中，我们已经为智能体开发了呼喊提前架构，并为该架构开发了一种混合学习方法，将强化学习与进化学习相结合。为游戏《Battle for Wesnoth》开发控制敌人单位的AI的概念验证系统表明，最优秀的AI能够击败游戏中相当优秀的人类创造的中央AI。我们探索了使用行为学习来测试系统弱点的几个应用程序，包括游戏。在所有这些应用程序中，我们的测试系统能够发现被测试系统中的几个弱点，包括效率和安全性弱点。在这些工作的基础上，我们将在两个案例研究中探索我们将行为的一般学习与基于学习的测试相结合的想法，一个是威斯诺战役系统的扩展，另一个是自动化的牲畜配送中心，作为物联网应用的一个例子。这项研究的预期结果是能够以比当前实践更低的成本创建定制的开放分布式系统，并且由于自动化而具有更高的质量。

The long-term vision for my research is to use machine learning to improve the development of distributed systems software. My key idea is to modify and extend a concept for learning of cooperative behaviour to include the results of a learning-based testing component that finds negative consequences of the learned cooperative behaviour, including problems with intended functionality, efficiency and security. ***To learn cooperative behaviour, we see the components of a distributed system as agents that can perform actions. The learning tries to create controls for these agents, so that they together achieve the functionality the whole group of agents is tasked with.***Using learning-based testing for finding negative consequences is an evolutionary learning process in which the testing component creates interaction sequences with the tested system and evaluates the sequences based on how near they come to a testing goal. The evaluation is used to learn sequencers that come nearer and nearer to creating the negative consequences the testing system is looking for.***The main research challenge is how to combine the focus of learning of cooperative behaviour on trying to deal with all possible situations the system can be in with the focus of learning-based testing on finding one particular sequence of situations revealing a particular problem of the tested system. Possible solutions of this challenge can target the way learning-based testing is performed, generalizing its results, but also modifying the general learner and the architecture with exception rules. In earlier work, we have developed the shout-ahead architecture for agents and a hybrid learning method for this architecture, combining reinforcement learning with evolutionary learning. A proof-of-concept system for developing control AIs for enemy units for the game Battle for Wesnoth showed that the best learned AIs are able to beat the rather good human-created central AI that comes with the game. We explored the usage of learning of behavior to test systems for weaknesses for several applications, including games. In all these applications, our test systems were able to find several weaknesses in the tested systems, including efficiency and security weaknesses.***Building on these works, we will explore our ideas for combining general learning of behaviour with learning-based testing in two case studies, an extension of the system for Battle for Wesnoth and an automated cattle distribution center as an example of Internet of Things applications. The anticipated result of this research is the ability to create customized open distributed systems at lower cost than current practise and with higher quality due to automization.