III: EAGER: Collaborative Research: A Community Experiment Platform for Reproducibility and Generalizability

科学方法的一个标志是，实验应该被描述得足够详细，以便可以重复，甚至可以概括。 这意味着可以在名义上相同的配置上重复结果，然后通过在新数据集上重放结果来概括结果，并查看它们如何随不同参数变化。 原则上，这对于计算实验来说应该比自然科学实验更容易，因为计算过程不仅可以自动化，而且计算系统也不会受到困扰生命科学的“生物变异”的影响。 不幸的是，现有技术还远远没有达到这个目标。 大多数计算实验仅在论文中非正式地指定，其中实验结果在图形标题中简要描述；产生结果的代码很少可用；配置参数的更改会以不可预见的方式导致结果。由于重要的科学发现往往是一系列较小、不太重要的步骤的结果，因此能够发布完整记录和可重复的结果对于推动科学发展是必要的。 虽然对可重复性和普遍性的关注几乎遍及所有自然科学、计算科学和社会科学领域，但没有任何一个领域将这种关注确定为研究工作的目标。犹他大学和纽约大学之间的这个合作项目由工具和基础设施组成，通过利用和扩展支持来源的科学工作流程系统提供的基础设施，支持共享、测试和重用科学实验和结果的过程。该项目探讨了三个关键的研究问题：（1）如何打包和出版可重复和可推广的科学成果概要。 (2) 用于探索、比较、重用结果或可能发现针对给定问题的更好方法的适当算法和接口是什么？ 3）如何帮助审稿人在给定时间/资源限制的情况下生成信息最丰富的实验。这项工作的预期结果是一个软件基础设施，它允许作者创建工作流程，对导出结果（包括使用的数据、配置参数集和底层软件）的计算过程进行编码，发布这些结果并将其连接到报告结果的出版物。测试人员（或审阅者）可以重复和验证结果、匿名提问以及修改实验条件。 想要在以前的工作基础上进行研究的研究人员能够搜索、重现、比较和分析实验和结果。该基础设施支持许多学科的科学家得出、发布和共享可重复的结果。这项研究的结果，包括开发的软件，将通过项目网站 (http://www.vistrails.org/index.php/RepeatabilityCentral) 提供。

A hallmark of the scientific method has been that experiments should be described in enough detail that they can be repeated and perhaps generalized. This implies the possibility of repeating results on nominally equal configurations and then generalizing the results by replaying them on new data sets, and seeing how they vary with different parameters. In principle, this should be easier for computational experiments than for natural science experiments, because not only can computational processes be automated but also computational systems do not suffer from the "biological variation" that plagues the life sciences. Unfortunately, the state of the art falls far short of this goal. Most computational experiments are specified only informally in papers, where experimental results are briefly described in figure captions; the code that produced the results is seldom available; and configuration parameters change results in unforeseen ways. Because important scientific discoveries are often the result of sequences of smaller, less significant steps, the ability to publish results that are fully documented and reproducible is necessary for advancing science. While concern about repeatability and generalizability cuts across virtually all natural, computational, and social science fields, no single field has identified this concern as a target of a research effort.This collaborative project between the University of Utah and New York University consists of tools and infrastructure that supports the process of sharing, testing and re-using scientific experiments and results by leveraging and extending the infrastructure provided by provenance-enabled scientific workflow systems. The project explores three key research questions: (1) How to package and publish compendia of scientific results that are reproducible and generalizable. (2) What are appropriate algorithms and interfaces for exploring, comparing, re-using the results or potentially discovering better approaches for a given problem? 3) How to aid reviewers to generate experiments that are most informative given a time/resource limit.An expected result of this work is a software infrastructure that allows authors to create workflows that encode the computational processes that derive the results (including data used, configuration parameters set, and underlying software), publish and connect these to publications where the results are reported. Testers (or reviewers) can repeat and validate results, ask questions anonymously, and modify experimental conditions. Researchers, who want to build upon previous works, are able to search, reproduce, compare and analyze experiments and results. The infrastructure supports scientists, in many disciplines, to derive, publish and share reproducible results. Results of this research, including developed software will be available via the project web site ( http://www.vistrails.org/index.php/RepeatabilityCentral).