CCRI: ENS: Evolving the Defects4J Benchmark and Infrastructure to Enable Sustained Innovation and Reproducibility in Software Engineering Research

软件工程研究的目标是提高贯穿现代技术的软件的质量。为了取得重大进展，使用真实的人工产物来获得经验结果的重复性和可比性，这在许多软件工程研究领域中是必不可少的，包括软件工程中的软件测试和调试、程序理解、软件进化和机器学习。Defects4J真实软件故障数据库和支持基础设施的开发是为了支持软件工程研究界，特别是为了进一步提高可重复性，使下游研究能够更快、更好地完成，并使研究人员摆脱(重新)开发实验基础设施的负担。它的重点是通过让Defects4J更广泛地提供并满足软件工程研究界日益增长的兴趣和功能要求，来实现软件工程研究中的持续创新和再现性。具体地说，这个项目的目标是：(1)基于社区反馈，扩大Defects4J中当前的构件集；(2)开发用于挖掘额外构件的全自动框架；(3)面向未来并将所有构件包装起来，以用于现代编译器和运行时环境；以及(4)提供全面的教程和模板，以支持研究人员和教育工作者。高度自动化和结构化的社区贡献途径将支持持续增长。此外，该项目还促进了关于缺陷模式、预测和预防、研究技术的适用性和局限性以及经验评估的最佳实践的科学知识。最后，该项目支持研究生和本科生的软件工程教育。基准和基础设施允许学生复制已发表的结果，试验和比较现有的工具和技术，并快速评估新想法。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Software engineering research aims to increase the quality of the software that pervades modern technology. To make significant advances, reproducibility and comparability of empirical results, using realistic artifacts, are essential in many software engineering research areas, including software testing and debugging, program comprehension, software evolution, and machine learning in software engineering. The Defects4J database of real software faults and a supporting infrastructure were developed to support the software engineering research community, in particular to further reproducibility, to enable faster and better completion of downstream research, and to free researchers from the burden of (re-)developing an experiment infrastructure.This project evolves and extends Defects4J, which has been widely adopted by the software engineering research community. It focuses on enabling sustained innovation and reproducibility in software engineering research by making Defects4J more broadly available and meeting the growing interest and feature requests from the software engineering research community. Specifically, this project aims to (1) grow the current set of artifacts in Defects4J, based on community feedback, (2) develop a fully automated framework for mining additional artifacts, (3) future-proof and containerize all artifacts for use with modern compilers and runtime environments, and (4) provide comprehensive tutorials and templates to support researchers and educators. A high degree of automation and structured pathways to community contributions will support sustained growth. Additionally, this project advances scientific knowledge about defect patterns, prediction, and prevention, applicability and limitations of research techniques, and best practices for empirical evaluations. Finally, this project supports software engineering education at the graduate and undergraduate level. The benchmark and infrastructure allow students to reproduce published results, experiment with, and compare, existing tools and techniques, and quickly evaluate new ideas.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.