EAGER: Policy Design for Reproducibility and Data Sharing in Computational Science

科学计算正在成为科学方法的绝对核心，但是在实验细节的交流和结果的验证方面非常宽松的做法正在导致可信度危机。在整个计算科学领域，研究界现在正在质疑传统的交流模式，并寻求实施科学知识转移的方法，使复制已发表的计算结果成为可能。这通常意味着将计算的所有细节--数据和代码--作为已发表的计算结果的基础，方便地提供给其他人。将数据和代码公开提供给其他人会引发无数问题，即如何以适当和有效的方式实现可重复研究的目标。科学期刊的要求对出版决策有着强大的影响力，也是最不被理解的。本提案旨在了解期刊政策的现状，即关于已发表计算结果的可重复性，以及期刊政策向采用数据和代码共享转变的潜在因素。计算科学研究的高度粒度性质为研究社区中政策的有效性提供了一个自然的实验，这些社区具有不同的伴随压力，例如数据和代码库大小，隐私和法律的障碍，资本密集度和仪器水平，以及工业合作，仅举几例。有效性的衡量标准可以确定，因为特定领域的期刊显示了一系列关于数据和代码共享的立场，从要求两者到完全忽略这个问题。这些发现反过来将为整个计算研究领域有效的数据和代码共享政策的指导方针的创建提供信息。此外，本研究将进行详细的案例研究，描述期刊用于进一步重复研究的成功方法。最后，该项目本身也将作为一个案例研究，公开发布其已发表结果的数据和代码，并分析如何最好地促进类似研究的可重复性。

Scientific computation is emerging as absolutely central to the scientific method, but the prevalence of very relaxed practices regarding the communication of experimental details and the validation of results is leading to a credibility crisis. Across the computational sciences, the research community is now questioning traditional modes of communication and seeking to implement methods for scientific knowledge transfer that make replication of published computational findings possible. This typically means making all details of the computations - the data and code - underlying published computational results conveniently available to others.Making data and code openly available raises myriad questions regarding appropriate and effective ways of reaching the goal ofreproducible research. The requirements of scientific journals exert a powerful influence on publishing decisions, and are also the least well-understood.This proposal seeks to build an understanding of the current state of journal policy regarding reproducibility of published computational results, and of the factors underlying journal policy changes toward the adoption of data and code sharing. The highly granular nature ofcomputational science research provides a natural experiment for the study of effectiveness of policies in communities with different attendant pressures such as data and codebase size, privacy and legal barriers, capital intensity and level of instrumentation, and industrial collaboration, to name a few. Measures of effectiveness can be ascertained since domain-specific journals show a spectrum of positions on data and code sharing, from requiring both to ignoring the issue altogether.These findings will in turn inform the creation of guidelines regarding effective data and code sharing policies across the computational research landscape. In addition, this research will conduct detailed case studies describing successful approaches journals have used to further reproducible research. Finally, this project will also act as a case study itself, with the open release of the data and code underlying its published results and analysis of how best to facilitate reproducibility for similarly situated research.