CI-EN: Collaborative Research: An Experimental Infrastructure and a Database of Real Faults to Foster Reproducibility in Software Engineering Research

CI-EN：协作研究：实验基础设施和真实故障数据库，以促进软件工程研究的可重复性

• 批准号: 1822251
• 负责人: Michael Ernst
• 金额: $ 26.81万
• 依托单位: University of Washington
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-15 至 2021-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1822251&HistoricalAwards=false
• 关键词: CI; EN; Collaborative; Research; Experimental

A core tenet of the scientific method is reproducibility of experiments. To reproduce an empirical study in software engineering research, it is necessary to use the same experimental subjects and equivalent experimental infrastructure. Reproducibility in software engineering research is an important topic, as is evident from the recent establishment of artifact evaluation committees and from publication of papers with conflicting results. Prior efforts to establish reusable artifacts for specific software engineering domains are based on artificial data, are too small, are not uniformly organized, and/or lack extensible infrastructure. This holds the software engineering research community back, hampers reproducibility and comparability, and reduces confidence in empirical results.This project promotes reproducibility and comparability in software engineering research, through a large-scale database of real software defects and a standardized infrastructure that enables controlled experiments. It enhances an existing prototype (Defects4J) by adding new artifacts, adding infrastructure for labeling and experimenting with the artifacts, and improving usability. Specifically, this project provides (1) the world's largest database of isolated and annotated real software defects, (2) an extensible mining infrastructure that supports automated defect mining and isolation, (3) an extensible experimental infrastructure that also includes a statistics and replication package component, and (4) tutorials and training materials for researchers and educators.This project advances software engineering research in three ways. (1) It enables researchers to perform more realistic experiments, using real instead of artificial defects. (2) It frees researchers from the burden of (re-)developing an experimental infrastructure, allowing them to focus on research ideas. (3) It fosters reproducibility and comparability by providing reusable artifacts and data sets. Furthermore, this project advances software engineering education at the undergraduate and graduate level: students can experiment with existing techniques, reproduce published results, or quickly evaluate new ideas.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

科学方法的核心原则是实验的可重复性。为了重现软件工程研究中的实证研究，需要使用相同的实验对象和等效的实验基础设施。软件工程研究中的可重复性是一个重要的话题，从最近成立的工件评估委员会和发表的结果相互矛盾的论文中可以明显看出这一点。先前为特定软件工程领域建立可重用工件的努力基于人工数据，规模太小，组织不统一，和/或缺乏可扩展的基础设施。这阻碍了软件工程研究社区的发展，阻碍了可重复性和可比性，并降低了对实证结果的信心。该项目通过真实软件缺陷的大规模数据库和支持受控实验的标准化基础设施，促进软件工程研究的可重复性和可比性。它通过添加新工件、添加用于标记和试验工件的基础设施以及提高可用性来增强现有原型 (Defects4J)。具体来说，该项目提供（1）世界上最大的隔离和注释的真实软件缺陷数据库，（2）支持自动缺陷挖掘和隔离的可扩展挖掘基础设施，（3）还包括统计和复制包组件的可扩展实验基础设施，以及（4）为研究人员和教育工作者提供的教程和培训材料。该项目通过三种方式推进软件工程研究。 (1)它使研究人员能够使用真实的缺陷而不是人为的缺陷来进行更真实的实验。 (2) 它使研究人员摆脱了（重新）开发实验基础设施的负担，使他们能够专注于研究想法。 (3) 通过提供可重用的工件和数据集来促进可重复性和可比性。 此外，该项目推进了本科生和研究生水平的软件工程教育：学生可以试验现有技术、重现已发表的结果或快速评估新想法。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。