Collaborative Research: Disciplinary Improvements: Repeto: Building a Network for Practical Reproducibility in Experimental Computer Science

Repeto项目将促进社区实践，使可重复性成为计算机科学主流研究和教育活动的一部分。该项目旨在了解计算机科学系统社区可重复性的成本/收益方程式，使可重复性可行或不可行的因素，以及使实验难以实际重复性的孤立因素(无论是技术因素还是面向用途的因素)。这一研究协调网络将开展一系列活动，从包装实验以具有成本效益的方法；在教学中使用可再生性；与通过会议和机构赞助的可再生性倡议合作；举办社区活动，强调重复或可复制的实验，如黑客马拉松、比赛或排名；培养可重复实验的储存库并监测其使用/复制情况；报告技术现状和支持实际可再现性的紧急要求。该提案的结果将是一系列可在开放平台上复制的计算机科学实验，通过相关指标了解它们在主流研究和教育活动中使用的程度和程度，以及一系列关于当前在计算机科学研究中主流使用可重复性的推动因素和障碍的报告。可复制的实验将使用平台可编程性工具创建，包括变色龙环境和相关软件，如CHI、Trovi和Jupyter笔记本。这种平台可编程性方法允许实验者以可重复和持久的方式表达复杂的实验拓扑。将平台可编程性与可执行笔记本相结合，将允许研究人员捕获完整的实验过程，供其他研究人员随后复制。该奖项由CEISE高级网络基础设施办公室颁发，由CEISE计算机和网络系统部门联合支持。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The Repeto project will foster community practices to make reproducibility a part of mainstream research and education activities in computer science. The project seeks to understand the cost/benefit equation of reproducibility for the computer science systems community, the factors that make reproducibility feasible or infeasible, as well as isolate factors (be they technical or usage oriented) that make practical reproducibility of experiments difficult. This research coordination network will develop a range of activities from teaching methodology for packaging experiments for cost-effective replication; using reproducibility in teaching; collaboration with reproducibility initiatives sponsored through conferences and institutions; community events emphasizing repeating or replicating experiments such as hackathons, competitions, or rankings; fostering repositories of replicable experiments and monitoring their usage/replication; to reporting on state of art and emergent requirements for the support of practical reproducibility. The outcomes of the proposal will be a collection of computer science experiments replicable on open platforms, an understanding of how much and to what extent they are used in mainstream research and education activities via relevant metrics, as well as a series of reports on current enablers and obstacles towards mainstream use of reproducibility in computer science research.Replicable experiments will be created using platform programmability tools including the Chameleon environment and associated software such as CHI, Trovi, and Jupyter notebooks. This platform programmability approach allows experimenters to express complex experimental topologies in repeatable and persistent ways. Combining platform programmability with executable notebooks will allow investigators to capture the full experimental process for subsequent replication by other researchers.This award by the CISE Office of Advanced Cyberinfrastructure is jointly supported by the CISE Computer and Networked Systems Division.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.