Collaborative Research: Disciplinary Improvements: Repeto: Building a Network for Practical Reproducibility in Experimental Computer Science

Repeto项目将促进社区实践，使可重复性成为计算机科学主流研究和教育活动的一部分。 该项目旨在了解计算机科学系统社区再现性的成本/效益方程，使再现性可行或不可行的因素，以及使实验的实际再现性困难的隔离因素（无论是技术还是面向使用）。 这一研究协调网络将开展一系列活动，从教学方法到包装实验以实现成本效益高的复制;在教学中使用可复制性;与通过会议和机构赞助的可复制性倡议合作;强调重复或复制实验的社区活动，如黑客马拉松、竞赛或排名;培养可复制实验的储存库并监测其使用/复制情况;报告最新技术水平和紧急要求，以支持实际重现性。该提案的成果将是一系列可在开放平台上复制的计算机科学实验，通过相关指标了解它们在主流研究和教育活动中的使用量和使用程度，以及一系列关于在计算机科学研究中主流使用再现性的当前推动因素和障碍的报告。将使用平台可编程工具创建可复制的实验，包括Chameleon环境和相关的软件，如CHI、Trovi和Cameryter笔记本。 这种平台可编程性方法允许实验者以可重复和持久的方式表达复杂的实验拓扑。将平台可编程性与可执行的笔记本相结合，将使研究人员能够捕获完整的实验过程，供其他研究人员随后复制。CISE高级网络基础设施办公室的这一奖项由CISE计算机和网络系统部门共同支持。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查进行评估，被认为值得支持的搜索.

The Repeto project will foster community practices to make reproducibility a part of mainstream research and education activities in computer science. The project seeks to understand the cost/benefit equation of reproducibility for the computer science systems community, the factors that make reproducibility feasible or infeasible, as well as isolate factors (be they technical or usage oriented) that make practical reproducibility of experiments difficult. This research coordination network will develop a range of activities from teaching methodology for packaging experiments for cost-effective replication; using reproducibility in teaching; collaboration with reproducibility initiatives sponsored through conferences and institutions; community events emphasizing repeating or replicating experiments such as hackathons, competitions, or rankings; fostering repositories of replicable experiments and monitoring their usage/replication; to reporting on state of art and emergent requirements for the support of practical reproducibility. The outcomes of the proposal will be a collection of computer science experiments replicable on open platforms, an understanding of how much and to what extent they are used in mainstream research and education activities via relevant metrics, as well as a series of reports on current enablers and obstacles towards mainstream use of reproducibility in computer science research.Replicable experiments will be created using platform programmability tools including the Chameleon environment and associated software such as CHI, Trovi, and Jupyter notebooks. This platform programmability approach allows experimenters to express complex experimental topologies in repeatable and persistent ways. Combining platform programmability with executable notebooks will allow investigators to capture the full experimental process for subsequent replication by other researchers.This award by the CISE Office of Advanced Cyberinfrastructure is jointly supported by the CISE Computer and Networked Systems Division.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.