Reproduction and Replication: A Case Study with Automatic Essay Scoring
Reproduction and Replication: A Case Study with Automatic Essay Scoring
复制标题
再现与复制：论文自动评分案例研究
DOI：
--
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
Çagri Çöltekin
中科院分区：
文献类型：
--
作者：
Eva;Çagri Çöltekin
与许多实验科学一样，实验的可重复性在 NLP 界越来越受到关注。本文介绍了我们对自动论文评分（AES）早期研究的复制工作，该研究用于确定多语言环境中第二语言学习者的熟练程度。我们提出了三组具有不同目标的实验。首先，根据 LREC 2020 REPROLANG 共享任务的规定，我们使用原作者在同一数据集上发布的代码重新运行原始 AES 系统。其次，我们使用不同的实现对相同的数据重复相同的实验。第三，我们在不同的数据集和不同的语言上测试原始系统。我们的大部分发现与原始论文的发现一致。然而，我们的结果与原始论文中提出的结果之间存在一些差异。我们详细报告并讨论这些差异。我们进一步探讨了与通过复制确认研究结果相关的一些要点，包括数据集的选择、报告和变异性的解释、适当的评估指标的使用以及提供代码和数据。我们还讨论了术语“再现”和“复制”之间的不同用途和差异，并且我们认为“再现”，即通过不同环境下的独立实验来确认结论，比精确复制已发表的值更有价值。
As in many experimental sciences, reproducibility of experiments has gained ever more attention in the NLP community. This paper presents our reproduction efforts of an earlier study of automatic essay scoring (AES) for determining the proficiency of second language learners in a multilingual setting. We present three sets of experiments with different objectives. First, as prescribed by the LREC 2020 REPROLANG shared task, we rerun the original AES system using the code published by the original authors on the same dataset. Second, we repeat the same experiments on the same data with a different implementation. And third, we test the original system on a different dataset and a different language. Most of our findings are in line with the findings of the original paper. Nevertheless, there are some discrepancies between our results and the results presented in the original paper. We report and discuss these differences in detail. We further go into some points related to confirmation of research findings through reproduction, including the choice of the dataset, reporting and accounting for variability, use of appropriate evaluation metrics, and making code and data available. We also discuss the varying uses and differences between the terms reproduction and replication, and we argue that reproduction, the confirmation of conclusions through independent experiments in varied settings is more valuable than exact replication of the published values.
自然语言处理中再现性的三个维度。
DOI：
--
发表时间：
2018
期刊：
LREC ... International Conference on Language Resources & Evaluation : [proceedings]. International Conference on Language Resources and Evaluation
影响因子：
--
作者：
Cohen,KBretonnel;Xia,Jingbo;Zweigenbaum,Pierre;Callahan,TiffanyJ;Hargraves,Orin;Goss,Foster;Ide,Nancy;Névéol,Aurélie;Grouin,Cyril;Hunter,LawrenceE
通讯作者：
Hunter,LawrenceE