Explorative Test Oracle Generation

探索性测试 Oracle 生成

• 批准号: EP/K030353/1
• 负责人: Gordon Fraser
• 金额: $ 11.81万
• 依托单位: University of Sheffield
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2014
• 资助国家: 英国
• 起止时间: 2014 至 无数据
• 项目状态: 已结题
• 来源: https://gtr.ukri.org/projects?ref=EP%2FK030353%2F1
• 关键词: Explorative; Test; Oracle; Generation

Testing is a crucial part of any software development process. Testing is also very expensive: Common estimations list the effort of software testing at 50% of the average budget. Our society increasingly depends on a working information infrastructure for more and more aspects of civic, commercial, and social life, while software at the same time becomes ever more complex. For example, a modern car has up to 100 million lines of software code, and software errors can easily lead to fatal consequences. Improving techniques to identify errors in software is therefore of utmost importance.Manual testing is common practice in software development. As manually testing a program is a laborious and error prone task, automation is desirable. However, automation requires the user to specify the correct behaviour up-front in terms of a specification, or later by adding test oracles to automatically generated tests - both alternatives are difficult. This problem is obliterated as test quality is usually measured with oracle-agnostic code coverage metrics. In truth, however, a test without a good oracle cannot find software bugs. This is the oracle problem, one of the longest standing and greatest remaining challenges in software testing.As both writing specifications and writing test oracles is difficult and needs to be done manually, this proposal aims to push automation further by exploring the middle ground: The novel concept of an oracle template allows to specify what should be tested and checked, but crucially, it does not require specifying the expected behaviour. Instead, automated test generation instantiates user-specified oracle templates to concrete tests with oracles, and the developer decides case by case about correctness. Thus, programs can be tested without the developer needing to write a specification or having to suffer through seemingly purposeless generated tests. Because test generation is driven by oracles, all tests have a purpose and the essential oracles required to be effective at finding software bugs.The novel concept of oracle templates requires extension of the current state of the art in test generation, as current techniques either assume the existence of an automated oracle (e.g. a specification) or focus exclusively on the code. This creates three challenges, which will be addressed in this project:-- Existing code-based testing techniques focus on reaching points in the code. This project will define the concept of oracle templates, and will explore test generation based on oracle templates as a search problem. Given an oracle template, search-based testing techniques will automatically create instances, which are test cases with oracles.-- Systematic testing is traditionally driven by the idea that a good test set covers all the code, which completely ignores the test oracle problem. This project will define systematic criteria and corresponding search-based test generation techniques to thoroughly test programs based on oracle templates. These criteria will ensure coverage of oracle templates, but will also ensure that the code is executed and checked by oracles (e.g. by applying mutation and data-flow analysis).-- It is impossible to take the human out of the software testing loop completely. Oracle templates are an attempt at minimizing the human effort, but the task of writing oracle templates still requires manual effort. Therefore, this project will explore strategies to automatically synthesise oracle templates based on standard testing patterns and usage examples. Ultimately, a developer would have all tests and oracles generated automatically on the click of a button, leaving only the task of confirming correctness of the produced examples.The success in addressing these challenges will be measured using automated experiments, controlled studies with student subjects, and industrial case studies at Google and Microsoft.

测试是任何软件开发过程中至关重要的一部分。测试也非常昂贵：通常的估算显示，软件测试的工作量占平均预算的50%。我们的社会越来越依赖于公民、商业和社会生活的越来越多方面的工作信息基础设施，同时软件也变得越来越复杂。例如，一辆现代汽车拥有多达1亿行的软件代码，软件错误很容易导致致命的后果。因此，改进识别软件错误的技术是至关重要的。手动测试是软件开发中的常见做法。由于手动测试程序是一项费力且容易出错的任务，因此自动化是可取的。然而，自动化要求用户预先根据规范指定正确的行为，或者稍后通过向自动生成的测试添加测试预言来指定正确的行为--这两种选择都很困难。这个问题被忽略了，因为测试质量通常是用与Oracle无关的代码复盖率度量来衡量的。然而，事实上，没有好的先知的测试无法发现软件错误。这是Oracle问题，是软件测试中持续时间最长也是最大的挑战之一。由于编写规范和编写测试Oracle都很困难，需要手动完成，因此该建议旨在通过探索中间立场来进一步推动自动化：Oracle模板的新概念允许指定应该测试和检查的内容，但关键是，它不需要指定预期的行为。相反，自动测试生成将用户指定的Oracle模板实例化为使用Oracle进行的具体测试，开发人员将逐一决定正确性。因此，可以测试程序，而开发人员不需要编写规范，也不需要经历看似无目的的生成测试。因为测试生成是由Oracle驱动的，所以所有测试都有一个目的，并且需要基本的Oracle才能有效地发现软件错误。Oracle模板的新概念需要在测试生成中扩展当前技术水平，因为当前的技术要么假设存在自动化的Oracle(例如规范)，要么只关注代码。这带来了三个挑战，将在本项目中解决：--现有的基于代码的测试技术侧重于达到代码中的点。这个项目将定义Oracle模板的概念，并将探索基于Oracle模板的测试生成作为搜索问题。给定Oracle模板，基于搜索的测试技术将自动创建实例，这些实例是Oracle的测试用例。--系统测试传统上是由好的测试集覆盖所有代码的想法驱动的，这完全忽略了测试Oracle的问题。该项目将定义系统标准和相应的基于搜索的测试生成技术，以彻底测试基于Oracle模板的程序。这些标准将确保甲骨文模板的覆盖面，但也将确保代码由甲骨文执行和检查(例如，通过应用突变和数据流分析)。--不可能将人完全排除在软件测试循环之外。Oracle模板试图将人工工作量降至最低，但编写Oracle模板的任务仍然需要手动完成。因此，该项目将探索基于标准测试模式和用例自动合成Oracle模板的策略。最终，开发人员只需点击一个按钮，就可以自动生成所有测试和先知，只留下确认生成的示例的正确性的任务。应对这些挑战的成功将通过自动化实验、学生受试者对照研究以及谷歌和微软的工业案例研究来衡量。

项目成果

Search-Based Software Engineering - 7th International Symposium, SSBSE 2015, Bergamo, Italy, September 5-7, 2015, Proceedings

基于搜索的软件工程 - 第七届国际研讨会，SSBSE 2015，意大利贝加莫，2015 年 9 月 5-7 日，会议记录

• DOI: 10.1007/978-3-319-22183-0_16
• 发表时间: 2015
• 作者: Campos J

Does Automated Unit Test Generation Really Help Software Testers? A Controlled Empirical Study

• DOI: 10.1145/2699688
• 发表时间: 2015-09
• 期刊: ACM Transactions on Software Engineering and Methodology (TOSEM)
• 作者: G. Fraser;Matthew Staats;Phil McMinn;Andrea Arcuri;F. Padberg

Generating TCP/UDP network data for automated unit test generation

生成 TCP/UDP 网络数据以自动生成单元测试

• DOI: 10.1145/2786805.2786828
• 发表时间: 2015
• 作者: Arcuri A

A Memetic Algorithm for whole test suite generation

• DOI: 10.1016/j.jss.2014.05.032
• 发表时间: 2015-05
• 期刊: J. Syst. Softw.
• 作者: G. Fraser;Andrea Arcuri;Phil McMinn

Automated unit test generation for classes with environment dependencies

• DOI: 10.1145/2642937.2642986
• 发表时间: 2014-09
• 期刊: Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering
• 作者: Andrea Arcuri;G. Fraser;Juan P. Galeotti