CI-EN: Collaborative Research: An Experimental Infrastructure and a Database of Real Faults to Foster Reproducibility in Software Engineering Research

CI-EN：协作研究：实验基础设施和真实故障数据库，以促进软件工程研究的可重复性

• 批准号: 1822251
• 负责人: Michael Ernst
• 金额: $ 26.81万
• 依托单位: University of Washington
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-15 至 2021-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1822251&HistoricalAwards=false
• 关键词: CI; EN; Collaborative; Research; Experimental

A core tenet of the scientific method is reproducibility of experiments. To reproduce an empirical study in software engineering research, it is necessary to use the same experimental subjects and equivalent experimental infrastructure. Reproducibility in software engineering research is an important topic, as is evident from the recent establishment of artifact evaluation committees and from publication of papers with conflicting results. Prior efforts to establish reusable artifacts for specific software engineering domains are based on artificial data, are too small, are not uniformly organized, and/or lack extensible infrastructure. This holds the software engineering research community back, hampers reproducibility and comparability, and reduces confidence in empirical results.This project promotes reproducibility and comparability in software engineering research, through a large-scale database of real software defects and a standardized infrastructure that enables controlled experiments. It enhances an existing prototype (Defects4J) by adding new artifacts, adding infrastructure for labeling and experimenting with the artifacts, and improving usability. Specifically, this project provides (1) the world's largest database of isolated and annotated real software defects, (2) an extensible mining infrastructure that supports automated defect mining and isolation, (3) an extensible experimental infrastructure that also includes a statistics and replication package component, and (4) tutorials and training materials for researchers and educators.This project advances software engineering research in three ways. (1) It enables researchers to perform more realistic experiments, using real instead of artificial defects. (2) It frees researchers from the burden of (re-)developing an experimental infrastructure, allowing them to focus on research ideas. (3) It fosters reproducibility and comparability by providing reusable artifacts and data sets. Furthermore, this project advances software engineering education at the undergraduate and graduate level: students can experiment with existing techniques, reproduce published results, or quickly evaluate new ideas.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

科学方法的核心原则是实验的可重复性。为了重现软件工程研究中的实证研究，有必要使用相同的实验对象和等效的实验基础设施。软件工程研究中的可复制性是一个重要的主题，这从最近建立的工件评估委员会和发表的具有相互矛盾的结果的论文中可以看出。为特定软件工程领域建立可重用工件的先前努力是基于人工数据的，太小，没有统一组织，和/或缺乏可扩展的基础设施。这阻碍了软件工程研究社区的发展，阻碍了可重复性和可比性，降低了对经验结果的信心。本项目通过建立一个包含真实的软件缺陷的大规模数据库和一个标准化的基础设施，使受控实验成为可能，从而促进了软件工程研究的可重复性和可比性。它通过添加新的工件、添加用于标记和试验工件的基础设施以及提高可用性来增强现有的原型（Defects 4J）。具体来说，该项目提供了（1）世界上最大的隔离和注释的真实的软件缺陷数据库，（2）支持自动缺陷挖掘和隔离的可扩展挖掘基础设施，（3）还包括统计和复制包组件的可扩展实验基础设施，（4）为研究人员和教育工作者提供的教程和培训材料。(1)它使研究人员能够进行更真实的实验，使用真实的而不是人为的缺陷。(2)它将研究人员从（重新）开发实验基础设施的负担中解放出来，使他们能够专注于研究思路。(3)它通过提供可重复使用的工件和数据集来促进可重复性和可比性。 此外，该项目推进了本科和研究生阶段的软件工程教育：学生可以实验现有技术，复制已发表的结果，或快速评估新想法。该奖项反映了NSF的法定使命，并被认为值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估来支持。