III: EAGER: Collaborative Research: A Community Experiment Platform for Reproducibility and Generalizability

科学方法的一个特点是，实验应该描述得足够详细，以便可以重复，也许还可以推广。这意味着在名义上相等的配置上重复结果的可能性，然后通过在新数据集上重播结果来推广结果，并查看它们如何随不同参数变化。原则上，计算实验应该比自然科学实验更容易，因为不仅计算过程可以自动化，而且计算系统不会遭受困扰生命科学的“生物变异”。不幸的是，目前的技术水平远远达不到这个目标。大多数计算实验只在非正式的论文中指定，实验结果在图标题中简要描述；产生结果的代码很少可用；配置参数的改变会导致无法预料的结果。由于重要的科学发现往往是一系列较小的、不太重要的步骤的结果，因此能够发表有完整记录和可重复的结果对于推进科学是必要的。尽管几乎所有自然科学、计算科学和社会科学领域都关注可重复性和泛化性，但没有一个领域将这一关注确定为研究工作的目标。犹他大学和纽约大学之间的这个合作项目由工具和基础设施组成，这些工具和基础设施通过利用和扩展由支持来源的科学工作流系统提供的基础设施来支持共享、测试和重用科学实验和结果的过程。该项目探讨了三个关键的研究问题：(1)如何包装和出版可重复和可推广的科学成果纲要。(2)对于探索、比较、重用结果或潜在地为给定问题发现更好的方法，什么是合适的算法和接口？3)在有限的时间/资源下，如何帮助审稿人生成最有信息量的实验。这项工作的预期结果是一个软件基础结构，它允许作者创建工作流，对导出结果（包括使用的数据、配置参数集和底层软件）的计算过程进行编码，发布并将这些过程连接到报告结果的出版物。测试人员（或审查人员）可以重复并验证结果，匿名提问，并修改实验条件。研究人员，谁想要建立在以前的工作，能够搜索，复制，比较和分析实验和结果。该基础设施支持许多学科的科学家推导、发表和分享可重复的结果。这项研究的结果，包括开发的软件将通过项目网站（http://www.vistrails.org/index.php/RepeatabilityCentral）提供。

A hallmark of the scientific method has been that experiments should be described in enough detail that they can be repeated and perhaps generalized. This implies the possibility of repeating results on nominally equal configurations and then generalizing the results by replaying them on new data sets, and seeing how they vary with different parameters. In principle, this should be easier for computational experiments than for natural science experiments, because not only can computational processes be automated but also computational systems do not suffer from the "biological variation" that plagues the life sciences. Unfortunately, the state of the art falls far short of this goal. Most computational experiments are specified only informally in papers, where experimental results are briefly described in figure captions; the code that produced the results is seldom available; and configuration parameters change results in unforeseen ways. Because important scientific discoveries are often the result of sequences of smaller, less significant steps, the ability to publish results that are fully documented and reproducible is necessary for advancing science. While concern about repeatability and generalizability cuts across virtually all natural, computational, and social science fields, no single field has identified this concern as a target of a research effort.This collaborative project between the University of Utah and New York University consists of tools and infrastructure that supports the process of sharing, testing and re-using scientific experiments and results by leveraging and extending the infrastructure provided by provenance-enabled scientific workflow systems. The project explores three key research questions: (1) How to package and publish compendia of scientific results that are reproducible and generalizable. (2) What are appropriate algorithms and interfaces for exploring, comparing, re-using the results or potentially discovering better approaches for a given problem? 3) How to aid reviewers to generate experiments that are most informative given a time/resource limit.An expected result of this work is a software infrastructure that allows authors to create workflows that encode the computational processes that derive the results (including data used, configuration parameters set, and underlying software), publish and connect these to publications where the results are reported. Testers (or reviewers) can repeat and validate results, ask questions anonymously, and modify experimental conditions. Researchers, who want to build upon previous works, are able to search, reproduce, compare and analyze experiments and results. The infrastructure supports scientists, in many disciplines, to derive, publish and share reproducible results. Results of this research, including developed software will be available via the project web site ( http://www.vistrails.org/index.php/RepeatabilityCentral).