III: EAGER: Collaborative Research: A Community Experiment Platform for Reproducibility and Generalizability

科学方法的一个标志是，实验应该被描述得足够详细，以便它们可以重复，甚至可以推广。 这意味着在名义上相等的配置上重复结果的可能性，然后通过在新的数据集上重放结果来概括结果，并查看它们如何随不同参数而变化。 原则上，这对于计算实验来说应该比自然科学实验更容易，因为不仅计算过程可以自动化，而且计算系统不会遭受困扰生命科学的“生物变异”。 不幸的是，最先进的福尔斯远远达不到这一目标。 大多数计算实验只是在论文中非正式地指定，实验结果在图标题中简要描述;产生结果的代码很少可用;配置参数以不可预见的方式改变结果。由于重要的科学发现往往是一系列较小、不太重要的步骤的结果，因此，发表有充分记录和可重复的结果的能力对于推进科学是必要的。 虽然对可重复性和可推广性的关注几乎贯穿了所有的自然科学、计算科学和社会科学领域，但没有一个领域将这种关注确定为研究工作的目标。犹他州大学和纽约大学之间的这个合作项目包括支持共享过程的工具和基础设施，通过利用和扩展来源支持的科学工作流程系统提供的基础设施，测试和重新使用科学实验和结果。该项目探讨了三个关键研究问题：（1）如何包装和出版可复制和可推广的科学成果纲要。(2)什么是适当的算法和接口，用于探索，比较，重用结果或潜在地发现给定问题的更好方法？3)如何帮助评审人员在给定时间/资源限制的情况下生成信息量最大的实验。这项工作的预期结果是一个软件基础设施，该软件基础设施允许作者创建工作流，该工作流对得出结果的计算过程进行编码（包括使用的数据，配置参数集和底层软件），将这些结果发布并连接到报告结果的出版物。测试人员（或评审人员）可以重复和验证结果，匿名提问，并修改实验条件。 研究人员，谁想要建立在以前的作品，能够搜索，复制，比较和分析实验和结果。该基础设施支持许多学科的科学家获得、发布和分享可复制的结果。这项研究的结果，包括开发的软件，将通过项目网站（http：//www.vistrails.org/index.php/RepeatabilityCentral）提供。

A hallmark of the scientific method has been that experiments should be described in enough detail that they can be repeated and perhaps generalized. This implies the possibility of repeating results on nominally equal configurations and then generalizing the results by replaying them on new data sets, and seeing how they vary with different parameters. In principle, this should be easier for computational experiments than for natural science experiments, because not only can computational processes be automated but also computational systems do not suffer from the "biological variation" that plagues the life sciences. Unfortunately, the state of the art falls far short of this goal. Most computational experiments are specified only informally in papers, where experimental results are briefly described in figure captions; the code that produced the results is seldom available; and configuration parameters change results in unforeseen ways. Because important scientific discoveries are often the result of sequences of smaller, less significant steps, the ability to publish results that are fully documented and reproducible is necessary for advancing science. While concern about repeatability and generalizability cuts across virtually all natural, computational, and social science fields, no single field has identified this concern as a target of a research effort.This collaborative project between the University of Utah and New York University consists of tools and infrastructure that supports the process of sharing, testing and re-using scientific experiments and results by leveraging and extending the infrastructure provided by provenance-enabled scientific workflow systems. The project explores three key research questions: (1) How to package and publish compendia of scientific results that are reproducible and generalizable. (2) What are appropriate algorithms and interfaces for exploring, comparing, re-using the results or potentially discovering better approaches for a given problem? 3) How to aid reviewers to generate experiments that are most informative given a time/resource limit.An expected result of this work is a software infrastructure that allows authors to create workflows that encode the computational processes that derive the results (including data used, configuration parameters set, and underlying software), publish and connect these to publications where the results are reported. Testers (or reviewers) can repeat and validate results, ask questions anonymously, and modify experimental conditions. Researchers, who want to build upon previous works, are able to search, reproduce, compare and analyze experiments and results. The infrastructure supports scientists, in many disciplines, to derive, publish and share reproducible results. Results of this research, including developed software will be available via the project web site ( http://www.vistrails.org/index.php/RepeatabilityCentral).