Reference-based Metrics can be Replaced with Reference-less Metrics in Evaluating Grammatical Error Correction Systems
Reference-based Metrics can be Replaced with Reference-less Metrics in Evaluating Grammatical Error Correction Systems
复制标题
在评估语法纠错系统时，基于引用的度量可以替换为无引用的度量
DOI：
--
复制
发表时间：
2017
期刊：
影响因子：
--
通讯作者：
Kentaro Inui
中科院分区：
文献类型：
--
作者：
Hiroki Asano;Tomoya Mizumoto;Kentaro Inui
在语法错误纠正（GEC）中，自动评估系统输出需要黄金标准的参考，必须手动创建，因此往往既昂贵又有限。为了解决这个问题，最近出现了一种无参考的方法;然而，以前的无参考的度量，只考虑语法性的标准，没有工作以及基于参考的度量。本研究探讨了扩展先前基于语法的方法来建立GEC系统的无参考评估方法的潜力。此外，我们的经验表明，一个参考少的指标，结合流畅性和意义的保留与语法提供了一个更好的估计手动分数比常用的参考为基础的指标。据我们所知，这是第一个研究，提供了经验证据，一个参考的指标可以取代基于参考的指标评估GEC系统。
In grammatical error correction (GEC), automatically evaluating system outputs requires gold-standard references, which must be created manually and thus tend to be both expensive and limited in coverage. To address this problem, a reference-less approach has recently emerged; however, previous reference-less metrics that only consider the criterion of grammaticality, have not worked as well as reference-based metrics. This study explores the potential of extending a prior grammaticality-based method to establish a reference-less evaluation method for GEC systems. Further, we empirically show that a reference-less metric that combines fluency and meaning preservation with grammaticality provides a better estimate of manual scores than that of commonly used reference-based metrics. To our knowledge, this is the first study that provides empirical evidence that a reference-less metric can replace reference-based metrics in evaluating GEC systems.