CRI: CI-EN: Collaborative Research: An Experimental Infrastructure and a Database of Real Faults to Foster Reproducibility in Software Engineering Research

科学方法的一个核心原则是实验的重复性。为了在软件工程研究中重现经验研究，有必要使用相同的实验对象和等效的实验基础设施。软件工程研究中的可重复性是一个重要的话题，从最近成立的构件评估委员会和发表的结果相互矛盾的论文中可以明显看出这一点。以前为特定软件工程领域建立可重用构件的工作是基于人工数据的，太小，没有统一的组织，和/或缺乏可扩展的基础设施。这阻碍了软件工程研究界的发展，阻碍了可重复性和可比性，并降低了对经验结果的信心。该项目通过大规模的真实软件缺陷数据库和支持受控实验的标准化基础设施来促进软件工程研究的可重复性和可比性。它通过添加新的构件、添加标记和试验构件的基础设施以及提高可用性来增强现有的原型(Defects4J)。具体地说，这个项目提供了(1)世界上最大的孤立和带注释的真实软件缺陷数据库，(2)支持自动缺陷挖掘和隔离的可扩展挖掘基础设施，(3)还包括统计和复制包组件的可扩展实验基础设施，以及(4)为研究人员和教育工作者提供的教程和培训材料。(1)它使研究人员能够进行更现实的实验，使用真实的缺陷而不是人为的缺陷。(2)它将研究人员从(重新)开发实验基础设施的负担中解放出来，使他们能够专注于研究想法。(3)通过提供可重复使用的构件和数据集，提高了可重复性和可比性。此外，该项目促进了本科生和研究生水平的软件工程教育：学生可以试验现有技术，复制已发表的结果，或快速评估新想法。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

A core tenet of the scientific method is reproducibility of experiments. To reproduce an empirical study in software engineering research, it is necessary to use the same experimental subjects and equivalent experimental infrastructure. Reproducibility in software engineering research is an important topic, as is evident from the recent establishment of artifact evaluation committees and from publication of papers with conflicting results. Prior efforts to establish reusable artifacts for specific software engineering domains are based on artificial data, are too small, are not uniformly organized, and/or lack extensible infrastructure. This holds the software engineering research community back, hampers reproducibility and comparability, and reduces confidence in empirical results.This project promotes reproducibility and comparability in software engineering research, through a large-scale database of real software defects and a standardized infrastructure that enables controlled experiments. It enhances an existing prototype (Defects4J) by adding new artifacts, adding infrastructure for labeling and experimenting with the artifacts, and improving usability. Specifically, this project provides (1) the world's largest database of isolated and annotated real software defects, (2) an extensible mining infrastructure that supports automated defect mining and isolation, (3) an extensible experimental infrastructure that also includes a statistics and replication package component, and (4) tutorials and training materials for researchers and educators.This project advances software engineering research in three ways. (1) It enables researchers to perform more realistic experiments, using real instead of artificial defects. (2) It frees researchers from the burden of (re-)developing an experimental infrastructure, allowing them to focus on research ideas. (3) It fosters reproducibility and comparability by providing reusable artifacts and data sets. Furthermore, this project advances software engineering education at the undergraduate and graduate level: students can experiment with existing techniques, reproduce published results, or quickly evaluate new ideas.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.