CRII: SHF: Machine-Learning-Based Test Effectiveness Prediction

测试有效性是软件测试的关键，它表明了测试在检测潜在软件缺陷方面的能力。更有效的测试可以检测到更多潜在的错误，从而有助于防止软件错误造成的经济损失甚至物理损害。因此，在过去的几十年里，大量的研究努力致力于测试有效性评估。近年来，突变检测作为一种通过计算人工注入细菌的检测率来衡量检测效果的强大方法，越来越受到学术界和产业界的关注。各种研究表明，突变测试产生的人工错误接近真实错误，证明了突变测试在测试效果评估中的有效性。然而，突变检测的一个主要障碍是效率问题？突变测试需要执行每个人工错误版本(即突变)，以检查测试套件是否可以检测到该错误，这是非常耗时的。因此，从各种开源项目中自动提取测试有效性信息(例如突变测试结果)，直接预测当前项目的测试有效性，而不需要任何突变执行，是一种轻量级但精确的测试有效性度量技术。更具体地说，PI建议基于故障检测的PIE理论收集的一套静态和动态特征来设计一个通用的分类框架。此外，本研究将探索高级程序分析、机器学习和软件挖掘技术的明智应用，以实现更强大的特征收集、更主动的学习以及更全面的训练数据准备。该方法将为使用各种编程语言和测试范例开发的项目带来高效而准确的测试效果评估，这对高质量的软件至关重要。此外，分类模型的培训将需要从大量开放源码项目收集各种基本测试、分析和挖掘信息，因此也可能使探索开放源码软件库的各种软件测试/分析/挖掘技术受益。

Test effectiveness, which indicates the capability of tests in detecting potential software bugs, is crucial for software testing. More effective tests can detect more potential bugs and thus help prevent economic loss or even physical damage caused by software bugs. Therefore, a huge body of research efforts have been dedicated to test effectiveness evaluation during the past decades. Recently, mutation testing, a powerful methodology that computes the detection rate of artificially injected bugs to measure test effectiveness, is drawing more and more attention from both the academia and industry. Various studies have shown that artificial bugs generated by mutation testing are close to real bugs, demonstrating mutation testing effectiveness in test effectiveness evaluation. However, a major obstacle for mutation testing is the efficiency problem ? mutation testing requires the execution of each artificial buggy version (i.e., mutant) to check whether the test suite can detect that bug, and which is extremely time consuming. Therefore, a light-weight but precise technique for measuring test effectiveness is highly desirable.The approach is to automatically extract test effectiveness information (e.g., mutation testing results) from various open-source projects to directly predict the test effectiveness of the current project without any mutant execution. More specifically, the PI proposes to design a general classification framework based on a suite of static and dynamic features collected according to the PIE theory of fault detection. Furthermore, this research will explore judicious applications of advanced program analysis, machine learning, and software mining techniques for more powerful feature collection, more active learning, as well as more comprehensive training data preparation. The proposed approach will result in efficient but precise test effectiveness evaluation for projects developed using various programming languages and test paradigms, which is crucial for high-quality software. Furthermore, the training of the classification models will require to collect various basic testing, analysis, and mining information from a huge number of open-source projects, and thus may also benefit a large variety of software testing/analysis/mining techniques that explore open-source software repositories.