Explorative Test Oracle Generation

测试是任何软件开发过程的关键部分。测试也是非常昂贵的：通常估计软件测试的工作量是平均预算的50%。我们的社会越来越多地依赖于信息基础设施的工作，因为公民，商业和社会生活的越来越多的方面，而软件同时变得越来越复杂。例如，一辆现代汽车拥有多达1亿行的软件代码，软件错误很容易导致致命的后果。因此，改进技术以识别软件中的错误是至关重要的。由于手动测试程序是一项费力且容易出错的任务，因此需要自动化。然而，自动化要求用户根据规范预先指定正确的行为，或者稍后通过向自动生成的测试添加测试预言机来指定正确的行为-这两种替代方案都很困难。由于测试质量通常是用与Oracle无关的代码覆盖度量来衡量的，因此这个问题就被消除了。然而，事实上，没有好的预言机的测试无法发现软件缺陷。这就是甲骨文问题，软件测试中存在时间最长、最大的挑战之一。由于编写规范和编写测试甲骨文都很困难，需要手动完成，因此本提案旨在通过探索中间地带来进一步推动自动化：甲骨文模板的新概念允许指定应该测试和检查的内容，但至关重要的是，它不需要指定预期的行为。相反，自动化测试生成将用户指定的Oracle模板实例化为使用Oracle的具体测试，开发人员根据具体情况决定正确性。因此，程序可以被测试，而不需要开发人员编写一个规范或必须通过看似无障碍的生成测试。由于测试生成是由甲骨文驱动的，所有的测试都有一个目的和必要的甲骨文需要有效地发现软件bugs.The甲骨文模板的新概念，需要在测试生成的最新技术的扩展，目前的技术要么假设存在一个自动化的甲骨文（例如规范）或专注于代码。这产生了三个挑战，这将在本项目中解决：-现有的基于代码的测试技术集中在代码中的到达点。这个项目将定义oracle模板的概念，并将探索基于oracle模板的测试生成作为一个搜索问题。给定一个Oracle模板，基于搜索的测试技术将自动创建实例，这些实例是使用Oracle的测试用例。传统上，系统测试是由一个好的测试集覆盖所有代码的想法驱动的，这完全忽略了测试预言的问题。这个项目将定义系统的标准和相应的基于搜索的测试生成技术，以彻底测试基于oracle模板的程序。这些标准将确保覆盖Oracle模板，但也将确保代码由Oracle执行和检查（例如通过应用突变和数据流分析）。把人完全排除在软件测试循环之外是不可能的。Oracle模板是最小化人工工作的尝试，但是编写Oracle模板的任务仍然需要手工工作。因此，本项目将探索基于标准测试模式和使用示例自动合成Oracle模板的策略。最终，开发人员只需点击一个按钮，就可以自动生成所有测试和预言，只需要确认生成的示例的正确性。解决这些挑战的成功将通过自动化实验、学生受试者的对照研究以及谷歌和微软的工业案例研究来衡量。

Testing is a crucial part of any software development process. Testing is also very expensive: Common estimations list the effort of software testing at 50% of the average budget. Our society increasingly depends on a working information infrastructure for more and more aspects of civic, commercial, and social life, while software at the same time becomes ever more complex. For example, a modern car has up to 100 million lines of software code, and software errors can easily lead to fatal consequences. Improving techniques to identify errors in software is therefore of utmost importance.Manual testing is common practice in software development. As manually testing a program is a laborious and error prone task, automation is desirable. However, automation requires the user to specify the correct behaviour up-front in terms of a specification, or later by adding test oracles to automatically generated tests - both alternatives are difficult. This problem is obliterated as test quality is usually measured with oracle-agnostic code coverage metrics. In truth, however, a test without a good oracle cannot find software bugs. This is the oracle problem, one of the longest standing and greatest remaining challenges in software testing.As both writing specifications and writing test oracles is difficult and needs to be done manually, this proposal aims to push automation further by exploring the middle ground: The novel concept of an oracle template allows to specify what should be tested and checked, but crucially, it does not require specifying the expected behaviour. Instead, automated test generation instantiates user-specified oracle templates to concrete tests with oracles, and the developer decides case by case about correctness. Thus, programs can be tested without the developer needing to write a specification or having to suffer through seemingly purposeless generated tests. Because test generation is driven by oracles, all tests have a purpose and the essential oracles required to be effective at finding software bugs.The novel concept of oracle templates requires extension of the current state of the art in test generation, as current techniques either assume the existence of an automated oracle (e.g. a specification) or focus exclusively on the code. This creates three challenges, which will be addressed in this project:-- Existing code-based testing techniques focus on reaching points in the code. This project will define the concept of oracle templates, and will explore test generation based on oracle templates as a search problem. Given an oracle template, search-based testing techniques will automatically create instances, which are test cases with oracles.-- Systematic testing is traditionally driven by the idea that a good test set covers all the code, which completely ignores the test oracle problem. This project will define systematic criteria and corresponding search-based test generation techniques to thoroughly test programs based on oracle templates. These criteria will ensure coverage of oracle templates, but will also ensure that the code is executed and checked by oracles (e.g. by applying mutation and data-flow analysis).-- It is impossible to take the human out of the software testing loop completely. Oracle templates are an attempt at minimizing the human effort, but the task of writing oracle templates still requires manual effort. Therefore, this project will explore strategies to automatically synthesise oracle templates based on standard testing patterns and usage examples. Ultimately, a developer would have all tests and oracles generated automatically on the click of a button, leaving only the task of confirming correctness of the produced examples.The success in addressing these challenges will be measured using automated experiments, controlled studies with student subjects, and industrial case studies at Google and Microsoft.