CRII: SHF: Understanding The Role of Software Test Adequacy Criteria in Search-Based Test Generation

软件测试确保了软件的健壮性和可靠性。由于测试人员不能先验地知道存在什么错误，因此已经提出了许多度量——从结构覆盖的度量到合成错误的检测——来判断测试用例的充分性。理论上，如果满足这些度量，测试应该足以检测故障。为了降低测试的高成本，可以使用优化算法自动生成测试套件。这些充分性度量非常适合指导自动化测试创建。然而，目前还没有一种充分性度量标准可以普遍地与“有效的故障检测”相对应。“测试人员面对的是令人眼花缭乱的测试选项，并且很少有关于何时使用一个标准而不是另一个标准的指导。这些度量是生成测试用例的坚实起点。在执行代码之前，无法检测到许多错误。然而，仅仅执行代码并不能确保充分的测试。代码的执行方式很重要。很明显，测试人员还不了解哪些充分性度量标准实际上与故障检测的高概率相对应，或者在什么情况下可以应用这些度量标准。因此，很明显，改进自动化测试生成需要更好地理解特定度量是有效的情况，隔离在这种情况下与故障检测相关的度量的特征，并建立和评估度量的使用和组合的指导方针——可能与特定的系统类型或域相关联——这将导致现实世界的故障检测。为了了解用于指导测试创建的标准的有效性和适用性，将对充分性标准和故障检测概率之间关系的本质进行大规模的实证调查。这项工作将对工业实践、软件工程教育以及——通过与工业伙伴和管理机构的传播和合作——公共安全和保障产生更广泛的影响。

Software testing ensures that software is robust and reliable. As testers cannot know what faults exist apriori, dozens of metrics---ranging from the measurement of structural coverage to the detection of synthetic faults---have been proposed to judge test case adequacy. In theory, if such metrics are fulfilled, tests should be adequate at detecting faults. To alleviate the high cost of testing, optimization algorithms can be used to automatically generate test suites. These adequacy metrics are well-suited for guiding automated test creation. However, there is no adequacy metric known to universally correspond to "effective fault detection.'' Testers are left with a bewildering number of testing options, and there is little guidance on when to use one criterion over another. These metrics are a solid starting point for test case generation. Many faults cannot be detected until the code has been executed. However, merely executing code does not ensure adequate testing. How code is executed is important. It is clear that testers do not yet understand which adequacy metrics actually correspond to a high probability of fault detection, or under what situations these metrics can be applied.Therefore, it is clear that improving automated test generation requires gaining a better understanding of the circumstances where particular metrics are effective, isolating the features of such metrics that correlate to fault detection in such circumstances, and establishing and evaluating guidelines for the use and combination of metrics - perhaps tied to particular system types or domains - that will result in real-world fault detection. Large-scale empirical investigations will be performed into the nature of the relationship between adequacy criteria and the probability of fault detection in order to understand the efficacy and applicability of the criteria that are used to guide test creation. This work will have broader impacts on industrial practices, software engineering education, and - through dissemination to and collaborations with industrial partners and regulatory agencies - public safety and security.