ReproHum: Investigating Reproducibility of Human Evaluations in Natural Language Processing

在过去的几个月里，我们已经为Reprofit项目奠定了基础（在工作计划文件的“项目前”一栏中进行了总结）：（i）对NLG中20年的人类评价进行了研究，详细审查并标记了171篇论文，（ii）开发了NLP评价的分类系统，（iii）提出了一项关于NLG中人类评价重现性的共同任务的建议，以及（iv）关于自然语言处理中人的评价的研讨会的建议。我们已经建立了一个由20个研究团队组成的国际网络，目前正在进行人体评估，他们将积极为该项目做出贡献（见跟踪记录部分），共同贡献超过80，000英镑。这个项目前活动为我们的工作创造了一个有利的起点，也意味着我们可以“立即开始”这项工作的科学核心。在这个基础项目中，我们的主要目标是开发一个方法框架，用于测试自然语言处理中人类评估的可重复性，以及一个多实验室范式，用于在实践中进行此类测试，这是NLP领域第一次进行此类研究。我们将（i）系统地诊断自然语言处理中人类评估再现性问题的程度，并调查相关的当前工作以解决它（WP 1）;（ii）发展自然语言处理中再现性测试的理论和方法基础（WP 2）;（iii）测试共享任务范式的适用性（在NLP领域普遍流行）用于再现性测试（WP 3）;（iv）创建多测试再现性研究的设计，并运行再现性研究，这是一项国际大规模多实验室工作，进行50多个个体，在过去10年中，NLP在人类评估方面的协调再现尝试（WP 4）;以及（v）通过技术会议和发展我们的国际研究人员网络，培养和建立关于如何解决再现性危机的国际共识（WP 5）。

Over the past few months, we have laid the groundwork for the ReproHum project (summarised in the 'pre-project' column in the Work Plan document) with (i) a study of 20 years of human evaluation in NLG which reviewed and labelled 171 papers in detail, (ii) the development of a classification system for NLP evaluations, (iii) a proposal for a shared task for reproducibility of human evaluation in NLG, and (iv) a proposal for a workshop on human evaluation in NLP. We have built an international network of 20 research teams currently working on human evaluation who will actively contribute to this project (see Track Record section), making combined contributions in kind of over £80,000. This pre-project activity has created an advantageous starting position for the proposed work, and means we can 'hit the ground running' with the scientifically interesting core of the work.In this foundational project, our key goals are the development of a methodological framework for testing the reproducibility of human evaluations in NLP, and of a multi-lab paradigm for carrying out such tests in practice, carrying out the first study of this kind in NLP. We will (i) systematically diagnose the extent of the human evaluation reproducibility problem in NLP and survey related current work to address it (WP1); (ii) develop the theoretical and methodological underpinnings for reproducibility testing in NLP (WP2); (iii) test the suitability of the shared-task paradigm (uniformly popular across NLP fields) for reproducibility testing (WP3); (iv) create a design for multi-test reproducibility studies, and run the ReproHum study, an international large-scale multi-lab effort conducting 50+ individual, coordinated reproduction attempts on human evaluations in NLP from the past 10 years (WP4); and (v) nurture and build international consensus regarding how to address the reproducibility crisis, via technical meetings and growing our international network of researchers (WP5).