Explorative Test Oracle Generation

测试是任何软件开发过程中至关重要的一部分。测试也是非常昂贵的：通常估计软件测试的工作量是平均预算的50%。我们的社会在公民、商业和社会生活的越来越多的方面越来越依赖于有效的信息基础设施，而与此同时软件也变得越来越复杂。例如，一辆现代汽车有多达1亿行软件代码，软件错误很容易导致致命的后果。因此，改进识别软件错误的技术是至关重要的。手工测试是软件开发中常见的实践。由于手动测试程序是一项费力且容易出错的任务，因此自动化是可取的。然而，自动化要求用户根据规范预先指定正确的行为，或者稍后通过添加测试oracle来自动生成测试——这两种选择都很困难。这个问题被消除了，因为测试质量通常是用与oracle无关的代码覆盖率度量来度量的。然而，事实上，没有好的oracle的测试是无法发现软件bug的。这就是oracle问题，是软件测试中存在时间最长、最大的挑战之一。由于编写规范和编写测试oracle都是困难的，需要手工完成，本建议旨在通过探索中间路线进一步推动自动化：oracle模板的新概念允许指定应该测试和检查的内容，但关键的是，它不需要指定预期的行为。相反，自动化的测试生成将用户指定的oracle模板实例化到使用oracle的具体测试中，并且开发人员逐个确定正确性。因此，开发人员无需编写规范，也不必忍受看似毫无目的的生成测试，就可以对程序进行测试。因为测试生成是由预言驱动的，所以所有的测试都有一个目的，以及有效发现软件bug所需要的基本预言。oracle模板的新概念需要扩展当前测试生成技术的状态，因为当前的技术要么假设存在自动化的oracle（例如规范），要么只关注代码。这产生了三个挑战，将在本项目中解决：—现有的基于代码的测试技术侧重于到达代码中的点。本项目将定义oracle模板的概念，并将基于oracle模板的测试生成作为一个搜索问题进行探索。给定一个oracle模板，基于搜索的测试技术将自动创建实例，这些实例是oracle的测试用例。—系统测试传统上是由一个好的测试集覆盖所有代码的想法驱动的，这完全忽略了测试oracle问题。该项目将定义系统的标准和相应的基于搜索的测试生成技术，以彻底测试基于oracle模板的程序。这些标准将确保oracle模板的覆盖率，但也将确保代码被oracle执行和检查（例如，通过应用突变和数据流分析）。——完全把人从软件测试循环中剔除是不可能的。Oracle模板是一种尽量减少人力的尝试，但是编写Oracle模板的任务仍然需要人力。因此，这个项目将探索基于标准测试模式和使用示例自动合成oracle模板的策略。最终，开发人员只需点击一个按钮就可以自动生成所有的测试和oracle，只需要确认生成的示例的正确性。解决这些挑战的成功将通过自动化实验、学生科目的对照研究以及b谷歌和微软的工业案例研究来衡量。

Testing is a crucial part of any software development process. Testing is also very expensive: Common estimations list the effort of software testing at 50% of the average budget. Our society increasingly depends on a working information infrastructure for more and more aspects of civic, commercial, and social life, while software at the same time becomes ever more complex. For example, a modern car has up to 100 million lines of software code, and software errors can easily lead to fatal consequences. Improving techniques to identify errors in software is therefore of utmost importance.Manual testing is common practice in software development. As manually testing a program is a laborious and error prone task, automation is desirable. However, automation requires the user to specify the correct behaviour up-front in terms of a specification, or later by adding test oracles to automatically generated tests - both alternatives are difficult. This problem is obliterated as test quality is usually measured with oracle-agnostic code coverage metrics. In truth, however, a test without a good oracle cannot find software bugs. This is the oracle problem, one of the longest standing and greatest remaining challenges in software testing.As both writing specifications and writing test oracles is difficult and needs to be done manually, this proposal aims to push automation further by exploring the middle ground: The novel concept of an oracle template allows to specify what should be tested and checked, but crucially, it does not require specifying the expected behaviour. Instead, automated test generation instantiates user-specified oracle templates to concrete tests with oracles, and the developer decides case by case about correctness. Thus, programs can be tested without the developer needing to write a specification or having to suffer through seemingly purposeless generated tests. Because test generation is driven by oracles, all tests have a purpose and the essential oracles required to be effective at finding software bugs.The novel concept of oracle templates requires extension of the current state of the art in test generation, as current techniques either assume the existence of an automated oracle (e.g. a specification) or focus exclusively on the code. This creates three challenges, which will be addressed in this project:-- Existing code-based testing techniques focus on reaching points in the code. This project will define the concept of oracle templates, and will explore test generation based on oracle templates as a search problem. Given an oracle template, search-based testing techniques will automatically create instances, which are test cases with oracles.-- Systematic testing is traditionally driven by the idea that a good test set covers all the code, which completely ignores the test oracle problem. This project will define systematic criteria and corresponding search-based test generation techniques to thoroughly test programs based on oracle templates. These criteria will ensure coverage of oracle templates, but will also ensure that the code is executed and checked by oracles (e.g. by applying mutation and data-flow analysis).-- It is impossible to take the human out of the software testing loop completely. Oracle templates are an attempt at minimizing the human effort, but the task of writing oracle templates still requires manual effort. Therefore, this project will explore strategies to automatically synthesise oracle templates based on standard testing patterns and usage examples. Ultimately, a developer would have all tests and oracles generated automatically on the click of a button, leaving only the task of confirming correctness of the produced examples.The success in addressing these challenges will be measured using automated experiments, controlled studies with student subjects, and industrial case studies at Google and Microsoft.