ReproHum: Investigating Reproducibility of Human Evaluations in Natural Language Processing

在过去的几个月里,我们已经奠定了基础ReproHum项目(项目前期工作的总结列工作计划文档中)与(i)研究20年的人类评价NLG详细审查和标签171篇论文,(ii)为NLP评估分类系统的发展,(iii)的提议共享任务再现性NLG人力评估,及(iv)人力评估研讨会NLP的提议。我们已经建立了一个由20个研究团队组成的国际网络，他们目前正在进行人类评估，他们将积极地为这个项目做出贡献（见跟踪记录部分），总捐款超过80,000英镑。这个项目前的活动为提议的工作创造了一个有利的起点，这意味着我们可以在工作的科学有趣核心上“落地运行”。在这个基础项目中，我们的主要目标是开发一种方法框架，用于测试NLP中人类评估的可重复性，以及在实践中进行此类测试的多实验室范例，在NLP中开展此类研究。我们将(i)系统地诊断NLP中人类评估可重复性问题的程度，并调查相关的当前工作以解决它（WP1）；（ii）发展NLP （WP2）可重复性测试的理论和方法基础；（iii）测试共享任务范式（在NLP领域普遍流行）用于可重复性测试的适用性（WP3）；（iv）创建一个多测试可重复性研究的设计，并运行rephum研究，这是一项国际大规模的多实验室努力，在过去10年中对NLP中的人类评估进行了50多个个体，协调的复制尝试（WP4）；(v)通过技术会议和发展我们的国际研究人员网络（WP5），培育和建立关于如何解决可重复性危机的国际共识。

Over the past few months, we have laid the groundwork for the ReproHum project (summarised in the 'pre-project' column in the Work Plan document) with (i) a study of 20 years of human evaluation in NLG which reviewed and labelled 171 papers in detail, (ii) the development of a classification system for NLP evaluations, (iii) a proposal for a shared task for reproducibility of human evaluation in NLG, and (iv) a proposal for a workshop on human evaluation in NLP. We have built an international network of 20 research teams currently working on human evaluation who will actively contribute to this project (see Track Record section), making combined contributions in kind of over £80,000. This pre-project activity has created an advantageous starting position for the proposed work, and means we can 'hit the ground running' with the scientifically interesting core of the work.In this foundational project, our key goals are the development of a methodological framework for testing the reproducibility of human evaluations in NLP, and of a multi-lab paradigm for carrying out such tests in practice, carrying out the first study of this kind in NLP. We will (i) systematically diagnose the extent of the human evaluation reproducibility problem in NLP and survey related current work to address it (WP1); (ii) develop the theoretical and methodological underpinnings for reproducibility testing in NLP (WP2); (iii) test the suitability of the shared-task paradigm (uniformly popular across NLP fields) for reproducibility testing (WP3); (iv) create a design for multi-test reproducibility studies, and run the ReproHum study, an international large-scale multi-lab effort conducting 50+ individual, coordinated reproduction attempts on human evaluations in NLP from the past 10 years (WP4); and (v) nurture and build international consensus regarding how to address the reproducibility crisis, via technical meetings and growing our international network of researchers (WP5).