CRII: SHF: Machine-Learning-Based Test Effectiveness Prediction

测试有效性是软件测试的关键，它反映了测试发现潜在软件缺陷的能力。更有效的测试可以发现更多潜在的错误，从而帮助防止软件错误造成的经济损失甚至物理损害。因此，在过去的几十年里，大量的研究工作一直致力于测试有效性评估。变异测试是一种通过计算人工注入错误的检测率来衡量测试有效性的方法，近年来受到学术界和工业界的广泛关注。各种研究表明，突变测试产生的人工bug接近真实的bug，证明了突变测试在测试有效性评估中的有效性。然而，突变测试的一个主要障碍是效率问题。变异测试需要执行每个人为的有缺陷的版本（即，突变）来检查测试套件是否可以检测到该bug，这是非常耗时的。因此，非常需要一种轻量级但精确的测试有效性测量技术。该方法是自动提取测试有效性信息（例如，突变测试结果），直接预测当前项目的测试有效性，无需任何突变执行。更具体地说，PI建议设计一个通用的分类框架的基础上收集的一套静态和动态功能，根据PIE理论的故障检测。此外，这项研究将探索先进的程序分析，机器学习和软件挖掘技术的明智应用，以实现更强大的特征收集，更主动的学习以及更全面的训练数据准备。所提出的方法将导致有效的，但精确的测试有效性评估的项目开发使用各种编程语言和测试范例，这是高质量的软件至关重要。此外，分类模型的训练将需要从大量的开源项目中收集各种基本的测试、分析和挖掘信息，因此也可能有利于探索开源软件存储库的各种软件测试/分析/挖掘技术。

Test effectiveness, which indicates the capability of tests in detecting potential software bugs, is crucial for software testing. More effective tests can detect more potential bugs and thus help prevent economic loss or even physical damage caused by software bugs. Therefore, a huge body of research efforts have been dedicated to test effectiveness evaluation during the past decades. Recently, mutation testing, a powerful methodology that computes the detection rate of artificially injected bugs to measure test effectiveness, is drawing more and more attention from both the academia and industry. Various studies have shown that artificial bugs generated by mutation testing are close to real bugs, demonstrating mutation testing effectiveness in test effectiveness evaluation. However, a major obstacle for mutation testing is the efficiency problem ? mutation testing requires the execution of each artificial buggy version (i.e., mutant) to check whether the test suite can detect that bug, and which is extremely time consuming. Therefore, a light-weight but precise technique for measuring test effectiveness is highly desirable.The approach is to automatically extract test effectiveness information (e.g., mutation testing results) from various open-source projects to directly predict the test effectiveness of the current project without any mutant execution. More specifically, the PI proposes to design a general classification framework based on a suite of static and dynamic features collected according to the PIE theory of fault detection. Furthermore, this research will explore judicious applications of advanced program analysis, machine learning, and software mining techniques for more powerful feature collection, more active learning, as well as more comprehensive training data preparation. The proposed approach will result in efficient but precise test effectiveness evaluation for projects developed using various programming languages and test paradigms, which is crucial for high-quality software. Furthermore, the training of the classification models will require to collect various basic testing, analysis, and mining information from a huge number of open-source projects, and thus may also benefit a large variety of software testing/analysis/mining techniques that explore open-source software repositories.