III: EAGER: Collaborative Research: A Community Experiment Platform for Reproducibility and Generalizability

科学方法的一个特点是，实验应该被描述得足够详细，以至于它们可以重复，或许还可以推广。这意味着有可能在名义上相等的配置上重复结果，然后通过在新的数据集上重播结果来推广结果，并查看它们如何随着不同的参数而变化。原则上，对于计算实验来说，这应该比自然科学实验更容易，因为不仅计算过程可以自动化，而且计算系统不会受到困扰生命科学的“生物变异”的影响。不幸的是，最先进的技术远远达不到这一目标。大多数计算实验只是在论文中非正式地指定的，实验结果在图形标题中简要描述；产生结果的代码很少可用；配置参数的变化以不可预见的方式产生结果。由于重要的科学发现往往是一系列较小、不太重要的步骤的结果，因此有能力发布完全有文件记录和可重现的结果，对于推进科学是必要的。虽然对重复性和概括性的关注几乎遍及所有自然、计算和社会科学领域，但没有一个领域将这一担忧确定为研究努力的目标。犹他大学和纽约大学的这个合作项目由工具和基础设施组成，通过利用和扩展支持起源的科学工作流系统提供的基础设施，支持共享、测试和重复使用科学实验和结果的过程。该项目探索了三个关键研究问题：(1)如何打包和出版可重复和可推广的科学成果纲要。(2)什么是合适的算法和界面，用于探索、比较、重复使用结果或潜在地发现针对给定问题的更好方法？3)如何在时间/资源限制的情况下帮助评审者生成信息量最大的实验。这项工作的预期结果是软件基础设施，允许作者创建工作流来编码得出结果的计算过程(包括使用的数据、配置参数集和底层软件)，并将这些发布和连接到报告结果的出版物。测试员(或审查员)可以重复和验证结果，匿名提问，并修改实验条件。研究人员想要在以前工作的基础上再接再厉，能够搜索、复制、比较和分析实验和结果。该基础设施支持许多学科的科学家得出、发表和分享可复制的结果。这项研究的结果，包括开发的软件，将通过项目网站(http://www.vistrails.org/index.php/RepeatabilityCentral).)提供

A hallmark of the scientific method has been that experiments should be described in enough detail that they can be repeated and perhaps generalized. This implies the possibility of repeating results on nominally equal configurations and then generalizing the results by replaying them on new data sets, and seeing how they vary with different parameters. In principle, this should be easier for computational experiments than for natural science experiments, because not only can computational processes be automated but also computational systems do not suffer from the "biological variation" that plagues the life sciences. Unfortunately, the state of the art falls far short of this goal. Most computational experiments are specified only informally in papers, where experimental results are briefly described in figure captions; the code that produced the results is seldom available; and configuration parameters change results in unforeseen ways. Because important scientific discoveries are often the result of sequences of smaller, less significant steps, the ability to publish results that are fully documented and reproducible is necessary for advancing science. While concern about repeatability and generalizability cuts across virtually all natural, computational, and social science fields, no single field has identified this concern as a target of a research effort.This collaborative project between the University of Utah and New York University consists of tools and infrastructure that supports the process of sharing, testing and re-using scientific experiments and results by leveraging and extending the infrastructure provided by provenance-enabled scientific workflow systems. The project explores three key research questions: (1) How to package and publish compendia of scientific results that are reproducible and generalizable. (2) What are appropriate algorithms and interfaces for exploring, comparing, re-using the results or potentially discovering better approaches for a given problem? 3) How to aid reviewers to generate experiments that are most informative given a time/resource limit.An expected result of this work is a software infrastructure that allows authors to create workflows that encode the computational processes that derive the results (including data used, configuration parameters set, and underlying software), publish and connect these to publications where the results are reported. Testers (or reviewers) can repeat and validate results, ask questions anonymously, and modify experimental conditions. Researchers, who want to build upon previous works, are able to search, reproduce, compare and analyze experiments and results. The infrastructure supports scientists, in many disciplines, to derive, publish and share reproducible results. Results of this research, including developed software will be available via the project web site ( http://www.vistrails.org/index.php/RepeatabilityCentral).