Artifact Description/Artifact Evaluation: A Reproducibility Bane or a Boon
Artifact Description/Artifact Evaluation: A Reproducibility Bane or a Boon
复制标题
工件描述/工件评估：可重复性的祸根或福音
DOI：
10.1145/3456287.3465479
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Malik, Tanu
中科院分区：
文献类型：
--
作者：
Malik, Tanu
一些系统研究会议现在将工件描述和工件评估（AD/AE）过程作为论文提交的一部分。被接受论文的作者可以选择提交大量的工件：文档、链接、工具、代码、数据和脚本，以独立验证其论文中的声明。工件评估委员会（AEC）评估工件，并用已接受的工件在论文上盖章，然后获得出版商徽章。AD/AE流程是否为作者和审稿人服务？对于像SCxy这样的大型会议，它是否具有可扩展性？以最后三个SCxy再现性计划为基础，本演讲将分析AD/AE过程的好处和痛苦。一些系统研究会议现在将工件描述和工件评估（AD/AE）过程作为论文提交的一部分。被接受论文的作者可以选择提交大量的工件：文档、链接、工具、代码、数据和脚本，以独立验证其论文中的声明。工件评估委员会（AEC）评估工件，并用已接受的工件在论文上盖章，然后获得出版商徽章。AD/AE流程是否为作者和审稿人服务？对于像SCxy这样的大型会议，它是否具有可扩展性？以最后三个SCxy再现性计划为基础，本演讲将分析AD/AE过程的好处和痛苦。我们将提出一种数据驱动的方法，使用调查结果来分析进行AD/AE过程中的技术和人员挑战。我们的方法将区分受益于AD的研究，即增加透明度与受益于AE的领域。AD/AE研究对象[1]提出了一组有趣的数据管理和系统挑战[2,3]。我们将深入研究研究对象，描述突出的特征，以及云基础设施、文档工作流和可重复的容器[4]如何简化一些AD/AE过程的握手。最后，我们将提出一个愿景，由此产生的策划，可重用的研究对象-这些研究对象如何成为未来几年推进计算可重复性和使可重复性评估实用的宝藏。
Several systems research conferences now incorporate an artifact description and artifact evaluation (AD/AE) process as part of the paper submission. Authors of accepted papers optionally submit a plethora of artifacts: documentation, links, tools, code, data, and scripts for independent validation of the claims in their paper. An artifact evaluation committee (AEC) evaluates the artifacts and stamps papers with accepted artifacts, which then receive publisher badges. Does this AD/AE process serve authors and reviewers? Is it scalable for large conferences such as SCxy? Using the last three SCxy Reproducibility Initiatives as the basis, this talk will analyze the benefits and the miseries of the AD/AE process. Several systems research conferences now incorporate an artifact description and artifact evaluation (AD/AE) process as part of the paper submission. Authors of accepted papers optionally submit a plethora of artifacts: documentation, links, tools, code, data, and scripts for independent validation of the claims in their paper. An artifact evaluation committee (AEC) evaluates the artifacts and stamps papers with accepted artifacts, which then receive publisher badges. Does this AD/AE process serve authors and reviewers? Is it scalable for large conferences such as SCxy? Using the last three SCxy Reproducibility Initiatives as the basis, this talk will analyze the benefits and the miseries of the AD/AE process.We will present a data-driven approach, using survey results to analyze technical and human challenges in conducting the AD/AE process. Our method will distinguish studies that benefit from AD, i.e., increased transparency versus areas that benefit from AE. The AD/AE research objects [1] present an interesting set of data management and systems challenges [2,3]. We will look under the hood of the research objects, describe prominent characteristics, and how cloud infrastructures, documented workflows, and reproducible containers [4] ease some of the AD/AE process hand-shakes. Finally, we will present a vision for the resulting curated, reusable research objects---how such research objects are a treasure in themselves for advancing computational reproducibility and making reproducible evaluation practical in the coming years.
Sciunits：可重复使用的研究对象
DOI：
10.1109/escience.2017.51
发表时间：
2017
期刊：
IEEE 13th International Conference on e-Science (e-Science
影响因子：
--
作者：
Ton That, Dai Hai;Fils, Gabriel;Yuan, Zhihao;Malik, Tanu
通讯作者：
Malik, Tanu