SGER: Learning Syntax-based Evaluation Metrics for Machine Translation

近年来，自动机器翻译系统取得了重大进展。缩短周转时间的一个有利因素是采用自动评估指标，这有助于指导开发人员改进翻译系统。然而，目前的指标是人类对系统输出判断的粗略近似值。SGER项目正在探索改善当前指标的方法。它在三个方面做到了这一点。首先，除了短语相似度之外，系统输出和人类参考文献之间的语法相似度的判断也被纳入度量。其次，机器学习技术正被应用于寻找与人类对质量判断相关的特征。第三，基于系统输出的语法和词汇特征，该项目正在确定对系统输出的错误和正确进行更细粒度的分析。在开发一个可学习的、语法敏感的度量标准时，判别方法将被推广到复杂的、结构化的问题中。一个信息量更大的自动评估指标将导致机器翻译技术的更快改进，使人们能够访问几个世纪前的档案和来自世界各地的文件。所有相关软件和非专有数据将被广泛分发。实验结果和发现的书面报告将通过学术出版物传播。

Recent years have witnessed significant improvements in automaticmachine translation systems. An enabling factor for this shortturnaround time is the adoption of automatic evaluation metrics, whichserve to guide developers to improve translation systems. However,current metrics are a coarse approximation of human judgments forsystem outputs. This SGER project is exploring ways of improvingcurrent metrics. It does so on three fronts. First, judgments ofsyntactic similarities between system outputs and human references arebeing incorporated into the metric in addition to phrasalsimilarities. Second, machine learning techniques are being appliedto find characteristics that are correlated to human judgments ofquality. Third, the project is determining more fine-grained analysesof what is wrong as well as what is right with system outputs based onsyntactic as well as lexical characteristics of the system outputs.In developing a learnable, syntax-aware metric, discriminative methodswill be generalized for complex, structured problems. A moreinformative automatic evaluation metric will lead to more rapidimprovements in machine translation technology, enabling access toarchives from centuries past and documents from around the world. Allrelevant software and non-proprietary data will be distributed widely.Written reports of experimental results and findings will bedisseminated through academic publications.