Collaborative Research: Disciplinary Improvements: Repeto: Building a Network for Practical Reproducibility in Experimental Computer Science

Repeto项目将促进社区实践，使再现性成为计算机科学主流研究和教育活动的一部分。该项目旨在了解计算机科学系统社区的可重复性的成本/收益方程，使可重复性可行或不可行的因素，以及隔离因素（无论是技术还是面向使用的），使实验的实际可重复性变得困难。这个研究协调网络将开展一系列活动，从包装实验的教学方法到具有成本效益的复制；在教学中运用再现性与通过会议和机构发起的可再现性倡议合作；强调重复或复制实验的社区活动，如黑客马拉松、竞赛或排名；建立可复制实验的存储库并监控其使用/复制；报告技术水平和紧急需求，以支持实际的可重复性。该提案的结果将是在开放平台上可复制的计算机科学实验的集合，通过相关指标了解它们在主流研究和教育活动中的使用程度和程度，以及一系列关于当前在计算机科学研究中主流使用可重复性的推动因素和障碍的报告。可复制的实验将使用平台可编程工具创建，包括变色龙环境和相关软件，如CHI， Trovi和Jupyter笔记本。这种平台可编程性方法允许实验人员以可重复和持久的方式表达复杂的实验拓扑。将平台可编程性与可执行笔记本相结合，将使研究人员能够捕捉到其他研究人员后续复制的完整实验过程。该奖项由CISE高级网络基础设施办公室颁发，并得到CISE计算机和网络系统部的联合支持。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The Repeto project will foster community practices to make reproducibility a part of mainstream research and education activities in computer science. The project seeks to understand the cost/benefit equation of reproducibility for the computer science systems community, the factors that make reproducibility feasible or infeasible, as well as isolate factors (be they technical or usage oriented) that make practical reproducibility of experiments difficult. This research coordination network will develop a range of activities from teaching methodology for packaging experiments for cost-effective replication; using reproducibility in teaching; collaboration with reproducibility initiatives sponsored through conferences and institutions; community events emphasizing repeating or replicating experiments such as hackathons, competitions, or rankings; fostering repositories of replicable experiments and monitoring their usage/replication; to reporting on state of art and emergent requirements for the support of practical reproducibility. The outcomes of the proposal will be a collection of computer science experiments replicable on open platforms, an understanding of how much and to what extent they are used in mainstream research and education activities via relevant metrics, as well as a series of reports on current enablers and obstacles towards mainstream use of reproducibility in computer science research.Replicable experiments will be created using platform programmability tools including the Chameleon environment and associated software such as CHI, Trovi, and Jupyter notebooks. This platform programmability approach allows experimenters to express complex experimental topologies in repeatable and persistent ways. Combining platform programmability with executable notebooks will allow investigators to capture the full experimental process for subsequent replication by other researchers.This award by the CISE Office of Advanced Cyberinfrastructure is jointly supported by the CISE Computer and Networked Systems Division.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.