CRI: CI-EN: Collaborative Research: An Experimental Infrastructure and a Database of Real Faults to Foster Reproducibility in Software Engineering Research

科学方法的核心原则是实验的可重复性。为了再现软件工程研究中的实证研究，有必要使用相同的实验对象和等效的实验基础设施。软件工程研究中的再现性是一个重要的主题，从最近建立的工件评估委员会和发表的结果相互矛盾的论文中可以明显看出这一点。先前为特定软件工程领域建立可重用工件的努力是基于人工数据的，规模太小，组织不统一，并且/或者缺乏可扩展的基础结构。这阻碍了软件工程研究界的发展，阻碍了再现性和可比性，并降低了对实证结果的信心。这个项目促进了软件工程研究的再现性和可比性，通过一个大规模的真实软件缺陷数据库和一个标准化的基础设施，使控制实验成为可能。它通过添加新的工件、添加用于标记和试验工件的基础设施以及改进可用性来增强现有的原型（缺陷4j）。具体来说，这个项目提供了(1)世界上最大的隔离和注释的真实软件缺陷数据库，(2)支持自动缺陷挖掘和隔离的可扩展挖掘基础设施，(3)可扩展的实验基础设施，也包括统计和复制包组件，以及(4)为研究人员和教育工作者提供的教程和培训材料。这个项目从三个方面推进了软件工程研究。它使研究人员能够进行更真实的实验，使用真实的而不是人为的缺陷。(2)它使研究人员从（重新）开发实验基础设施的负担中解脱出来，使他们能够专注于研究思路。(3)它通过提供可重用的工件和数据集来促进再现性和可比性。此外，这个项目推进了本科和研究生阶段的软件工程教育：学生可以用现有的技术进行实验，重现已发表的结果，或者快速评估新的想法。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

A core tenet of the scientific method is reproducibility of experiments. To reproduce an empirical study in software engineering research, it is necessary to use the same experimental subjects and equivalent experimental infrastructure. Reproducibility in software engineering research is an important topic, as is evident from the recent establishment of artifact evaluation committees and from publication of papers with conflicting results. Prior efforts to establish reusable artifacts for specific software engineering domains are based on artificial data, are too small, are not uniformly organized, and/or lack extensible infrastructure. This holds the software engineering research community back, hampers reproducibility and comparability, and reduces confidence in empirical results.This project promotes reproducibility and comparability in software engineering research, through a large-scale database of real software defects and a standardized infrastructure that enables controlled experiments. It enhances an existing prototype (Defects4J) by adding new artifacts, adding infrastructure for labeling and experimenting with the artifacts, and improving usability. Specifically, this project provides (1) the world's largest database of isolated and annotated real software defects, (2) an extensible mining infrastructure that supports automated defect mining and isolation, (3) an extensible experimental infrastructure that also includes a statistics and replication package component, and (4) tutorials and training materials for researchers and educators.This project advances software engineering research in three ways. (1) It enables researchers to perform more realistic experiments, using real instead of artificial defects. (2) It frees researchers from the burden of (re-)developing an experimental infrastructure, allowing them to focus on research ideas. (3) It fosters reproducibility and comparability by providing reusable artifacts and data sets. Furthermore, this project advances software engineering education at the undergraduate and graduate level: students can experiment with existing techniques, reproduce published results, or quickly evaluate new ideas.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.