EAGER: Cyberinfrastructure Reproducibility Project: Computational Science and Engineering

今天的科学进步在很大程度上依赖于计算机的使用，从实验室工作站到国家超级计算机。该项目面临的问题是确保源自计算研究的科学知识符合重复性和严谨性的高标准。NSF CISE理事会“亲爱的同事信：鼓励计算和通信研究中的重现性”(NSF 17-022,2016)鼓励研究人员在开发严格的协议时拥抱完整性和透明度。再现性问题通常关注通过开放数据、代码和其他研究对象实现的透明度。然而，提供再次运行所有分析的数据和代码是最低标准。在高性能计算的背景下，研究使用数百万美元的设备，仍然缺乏对不可再生性来源的了解。该项目使用计算流体力学以前研究的有条不紊的复制作为达成这一理解的模式，并制定研究设计的指导原则，以尽可能保证研究结果的重复性。该项目的成果将服务于NSF的使命，即促进科学进步；促进国家健康、繁荣和福利；通过对如何实现严格、可重复的计算研究带来新的和必要的理解来保障国防安全。该项目对已发表的计算流体力学研究进行了有条不紊的复制，作为计算科学和工程广泛领域的模型。其目的是1)识别和表征不可重复性的来源，以及2)制定可重复性设计指南，即保证研究结果可重复性的研究设计。该项目涉及科学软件库的作用(如线性代数解算器)、硬件架构的影响、可重复研究的新技术的作用(如容器、云服务)。该项目还为在高性能计算的背景下对复制研究的成功进行评估提供了指导方针，包括在何时合理或不合理地期望逐位数字重现。这项研究涉及计算研究的三个阶段：研究方法、研究交流和研究评估。该项目还开发了关于可重复计算研究的开放许可的培训材料，包括研究生研讨会课程和负责任的研究行为(RCR)模块。

The progress of science today relies heavily on the use of computers, from lab workstations to national supercomputers. This project confronts the issue of ensuring that scientific knowledge stemming from computational research complies to high standards of reproducibility and rigor. The NSF CISE Directorate "Dear Colleague Letter: Encouraging Reproducibility in Computing and Communications Research" (NSF 17-022, 2016) encourages researchers to embrace completeness and transparency in developing rigorous protocols. Reproducibility concerns often focus on transparency via open data, code, and other research objects. But providing the data and code to run all analyses again is, however, a minimum standard. In the context of high-performance computing where research uses multi-million-dollar facilities the understanding of where lie the sources of non-reproducibility is still lacking. This project uses methodical replication of previous studies in computational fluid dynamics as the model for reaching that understanding and for developing the guiding principles of study design that can guarantee, as much as possible, reproducible findings. The results of this project will serve NSF's mission to promote the progress of science; to advance the national health, prosperity and welfare; to secure the national defense by bringing new and necessary understanding about how to achieve rigorous, reproducible computational research. This project conducts methodical replication of published studies in computational fluid dynamics, as a model for the broad field of computational science and engineering. Its aims are to 1) identify and characterize sources of non-reproducibility, and 2) develop guidelines on Design for Reproducibility, i.e., study design guaranteeing reproducible findings. The project addresses the role of scientific software libraries (like linear algebra solvers), the influence of hardware architectures , the role of new technologies for reproducible research (e.g., containers, cloud services). The project also provides guidelines for making assessments about the success of replication studies in the context of high-performance computing, including when it is reasonable or not to expect bit-by-bit numerical reproducibility. The research tackles three phases of computational research: Research methods; Research communication; Research assessment. The project also develops openly licensed training materials on reproducible computational research, including a graduate seminar course and a Responsible Conduct of Research (RCR) module.