CRII: SHF: Machine-Learning-Based Test Effectiveness Prediction

测试有效性，即测试检测潜在软件缺陷的能力，对软件测试至关重要。更有效的测试可以检测到更多潜在的错误，从而帮助防止由软件错误造成的经济损失甚至物理损害。因此，在过去的几十年里，大量的研究工作致力于测试有效性的评估。近年来，突变检测作为一种通过计算人工注入病菌的检出率来衡量检测效果的强大方法，越来越受到学术界和工业界的关注。各种研究表明，突变测试产生的人工bug接近真实bug，证明了突变测试在测试有效性评价中的有效性。然而，突变检测的一个主要障碍是效率问题。突变测试需要执行每个人工错误版本（即，突变）来检查测试套件是否可以检测到该错误，这是非常耗时的。因此，非常需要一种轻量级但精确的技术来测量测试的有效性。该方法是自动从各种开源项目中提取测试有效性信息（例如，突变测试结果），直接预测当前项目的测试有效性，而不需要任何突变的执行。更具体地说，PI提出基于故障检测的PIE理论收集的一套静态和动态特征来设计一个通用的分类框架。此外，本研究将探索先进的程序分析、机器学习和软件挖掘技术的明智应用，以实现更强大的特征收集、更主动的学习以及更全面的训练数据准备。所提出的方法将导致对使用各种编程语言和测试范例开发的项目进行有效而精确的测试有效性评估，这对于高质量的软件是至关重要的。此外，分类模型的训练将需要从大量的开源项目中收集各种基本的测试、分析和挖掘信息，因此也可能有利于探索开源软件存储库的各种软件测试/分析/挖掘技术。

Test effectiveness, which indicates the capability of tests in detecting potential software bugs, is crucial for software testing. More effective tests can detect more potential bugs and thus help prevent economic loss or even physical damage caused by software bugs. Therefore, a huge body of research efforts have been dedicated to test effectiveness evaluation during the past decades. Recently, mutation testing, a powerful methodology that computes the detection rate of artificially injected bugs to measure test effectiveness, is drawing more and more attention from both the academia and industry. Various studies have shown that artificial bugs generated by mutation testing are close to real bugs, demonstrating mutation testing effectiveness in test effectiveness evaluation. However, a major obstacle for mutation testing is the efficiency problem ? mutation testing requires the execution of each artificial buggy version (i.e., mutant) to check whether the test suite can detect that bug, and which is extremely time consuming. Therefore, a light-weight but precise technique for measuring test effectiveness is highly desirable.The approach is to automatically extract test effectiveness information (e.g., mutation testing results) from various open-source projects to directly predict the test effectiveness of the current project without any mutant execution. More specifically, the PI proposes to design a general classification framework based on a suite of static and dynamic features collected according to the PIE theory of fault detection. Furthermore, this research will explore judicious applications of advanced program analysis, machine learning, and software mining techniques for more powerful feature collection, more active learning, as well as more comprehensive training data preparation. The proposed approach will result in efficient but precise test effectiveness evaluation for projects developed using various programming languages and test paradigms, which is crucial for high-quality software. Furthermore, the training of the classification models will require to collect various basic testing, analysis, and mining information from a huge number of open-source projects, and thus may also benefit a large variety of software testing/analysis/mining techniques that explore open-source software repositories.