SGER: Learning Syntax-based Evaluation Metrics for Machine Translation

近年来，自动机器翻译系统取得了显著的进步. 这种短周转时间的一个有利因素是采用了自动评估指标，用于指导开发人员改进翻译系统。 然而，目前的指标是一个粗略的近似人类的判断系统输出。 这个SGER项目正在探索改进当前指标的方法。 它在三个方面这样做。 首先，除了短语相似性之外，系统输出和人类参考之间的句法相似性的判断也被纳入到度量中。 其次，机器学习技术正在被应用于发现与人类对质量判断相关的特征。 第三，该项目将根据系统输出的语法和词汇特征，确定系统输出的错误和正确的更细粒度的分析。在开发一个可学习的、语法感知的度量标准时，将把判别方法推广到复杂的、结构化的问题。 一个更具创新性的自动评估指标将导致机器翻译技术的更快改进，使人们能够访问过去几个世纪的档案和世界各地的文件。 所有相关的软件和非专利数据将被广泛分发。实验结果和发现的书面报告将通过学术出版物传播。

Recent years have witnessed significant improvements in automaticmachine translation systems. An enabling factor for this shortturnaround time is the adoption of automatic evaluation metrics, whichserve to guide developers to improve translation systems. However,current metrics are a coarse approximation of human judgments forsystem outputs. This SGER project is exploring ways of improvingcurrent metrics. It does so on three fronts. First, judgments ofsyntactic similarities between system outputs and human references arebeing incorporated into the metric in addition to phrasalsimilarities. Second, machine learning techniques are being appliedto find characteristics that are correlated to human judgments ofquality. Third, the project is determining more fine-grained analysesof what is wrong as well as what is right with system outputs based onsyntactic as well as lexical characteristics of the system outputs.In developing a learnable, syntax-aware metric, discriminative methodswill be generalized for complex, structured problems. A moreinformative automatic evaluation metric will lead to more rapidimprovements in machine translation technology, enabling access toarchives from centuries past and documents from around the world. Allrelevant software and non-proprietary data will be distributed widely.Written reports of experimental results and findings will bedisseminated through academic publications.