CCRI: ENS: Evolving the Defects4J Benchmark and Infrastructure to Enable Sustained Innovation and Reproducibility in Software Engineering Research

软件工程研究的目的是提高现代技术中软件的质量。为了取得重大进展，使用现实的人工制品的经验结果的再现性和可比性，在许多软件工程研究领域是必不可少的，包括软件测试和调试，程序理解，软件进化，以及软件工程中的机器学习。开发了包含真实软件故障和支持性基础设施的Defects4J数据库，以支持软件工程研究界，特别是进一步提高再现性，以便更快更好地完成下游研究，并使研究人员摆脱（重新）开发实验基础设施的负担。该项目发展并扩展了软件工程研究社区广泛采用的Defects4J。它的重点是通过使缺陷4j更广泛地可用，并满足软件工程研究社区日益增长的兴趣和特性要求，从而支持软件工程研究中的持续创新和可再现性。具体来说，这个项目的目标是(1)基于社区反馈，在Defects4J中增加当前的工件集；(2)开发一个完全自动化的框架，用于挖掘额外的工件；(3)面向未来，并将所有工件容器化，以便与现代编译器和运行时环境一起使用；(4)提供全面的教程和模板，以支持研究人员和教育工作者。高度自动化和结构化的社区贡献途径将支持持续增长。另外，这个项目推进了关于缺陷模式、预测和预防、研究技术的适用性和局限性以及经验评估的最佳实践的科学知识。最后，这个项目支持研究生和本科生的软件工程教育。基准和基础设施允许学生重现已发布的结果，实验和比较现有的工具和技术，并快速评估新的想法。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Software engineering research aims to increase the quality of the software that pervades modern technology. To make significant advances, reproducibility and comparability of empirical results, using realistic artifacts, are essential in many software engineering research areas, including software testing and debugging, program comprehension, software evolution, and machine learning in software engineering. The Defects4J database of real software faults and a supporting infrastructure were developed to support the software engineering research community, in particular to further reproducibility, to enable faster and better completion of downstream research, and to free researchers from the burden of (re-)developing an experiment infrastructure.This project evolves and extends Defects4J, which has been widely adopted by the software engineering research community. It focuses on enabling sustained innovation and reproducibility in software engineering research by making Defects4J more broadly available and meeting the growing interest and feature requests from the software engineering research community. Specifically, this project aims to (1) grow the current set of artifacts in Defects4J, based on community feedback, (2) develop a fully automated framework for mining additional artifacts, (3) future-proof and containerize all artifacts for use with modern compilers and runtime environments, and (4) provide comprehensive tutorials and templates to support researchers and educators. A high degree of automation and structured pathways to community contributions will support sustained growth. Additionally, this project advances scientific knowledge about defect patterns, prediction, and prevention, applicability and limitations of research techniques, and best practices for empirical evaluations. Finally, this project supports software engineering education at the graduate and undergraduate level. The benchmark and infrastructure allow students to reproduce published results, experiment with, and compare, existing tools and techniques, and quickly evaluate new ideas.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.