Continuous Integration for Reproducible Shared Tasks with TIRA.io
Continuous Integration for Reproducible Shared Tasks with TIRA.io
复制标题
与 TIRA.io 持续集成可重复的共享任务
DOI：
--
复制
发表时间：
2023
期刊：
影响因子：
--
通讯作者：
Martin Potthast
中科院分区：
文献类型：
--
作者：
Maik Fröbe;Matti Wiegmann;Nikolay Kolyada;B. Grahm;Theresa Elstner;F. Loebe;Matthias Hagen;Benno Stein;Martin Potthast
大多数共同任务产生长期影响的一个主要障碍是缺乏可重复性。通常只出版组织者和参与者的试题集和论文。想要独立评估其他数据任务的最新技术水平的第三方必须重新实现参与者的软件。为从参与共同任务的人员那里收集软件而开发的工具在提交时只能部分验证软件的可靠性，更不用说长期验证了，而且无法使第三方在以后重新使用软件。我们已经彻底改革了TIRA综合研究架构，以解决所有这些问题。新版本简化了组织者的任务设置和参与者的软件提交，从本地计算机扩展到云端，支持按需资源分配到并行CPU和GPU处理，并只需几行代码即可导出本地复制。这是通过使用Git、Docker和Kubernetes实现具有行业标准持续集成和部署（CI/CD）管道的TIRA协议来实现的。
A major obstacle to the long-term impact of most shared tasks is their lack of reproducibility. Often only the test collections and the papers of the organizers and participants are published. Third parties who want to independently evaluate the state of the art for a task on other data must re-implement the participants’ software. The tools developed to collect software from participants in shared tasks only partially verify its reliability at the time of submission, much less long-term, and do not enable third parties to reuse it later. We have overhauled the TIRA Integrated Research Architecture to address all of these issues. The new version simplifies task setup for organizers and software submission for participants, scales from a local computer to the cloud, supports on-demand resource allocation up to parallel CPU and GPU processing, and enables export for local reproduction with just a few lines of code. This is achieved by implementing the TIRA protocol with an industry-standard continuous integration and deployment (CI/CD) pipeline using Git, Docker, and Kubernetes.