A Framework for Learning High Accuracy Evaluation Metrics for NLP Applications

这个项目研究了一种新的动态、自适应的方法来构建适用于各种自然语言处理应用的评估指标和方法，重点是机器翻译和摘要。主要目标是建立一个通用的框架，该框架可以很容易地支持为各种特定的NLP任务构建自动评估度量标准，并基于各种质量标准。对于给定的NLP任务(如机器翻译)和一组已建立的质量标准，该框架支持学习一组参数，这些参数产生与所需质量标准具有最佳相关性的“实例”评估度量。对于不同的任务或不同的质量标准集，可以通过使用由系统产生的输出、相同源数据的人的质量参考输出和系统输出的人的质量判断组成的可用的训练数据，通过快速训练过程来完成训练新的“实例”度量。新框架的一个强大的新创新是它能够使用被比较的两个字符串的所有重叠子序列的集合(也称为“跳过ngram”)。跳过n元语法匹配的过程通过强大的词到词对齐算法来增强，该算法预先限制跳过n元语法匹配的集合，同时允许词形变体、同义词或其他相关的单词之间的匹配。此外，我们的框架使用了一个基础良好的参数化模型来建立应该分配给每个检测到的重叠子序列的权值重要性，并且可以在检测匹配跳过图的过程中作为一个完整的过程来计算这些权重。其结果是一个极其强大的“公制生成”框架。在这个框架下，该项目将为机器翻译、摘要和其他NLP任务产生(实例化)特定的指标，这些指标更健壮、更敏感，并且与人类判断具有高度的相关性。该项目还探索了减少我们的结果指标对人类判断的依赖的方法。评估框架和针对特定任务的培训指标将向NLP研究社区公开提供。自动评估方法的影响不仅仅是为NLP任务提供了一种灵活的绩效衡量机制。我们希望我们的工作能够为各种跨语言应用程序中的特定任务定制评估指标，这将显著提高这些应用程序的整体性能。

This project investigates a new dynamic, adaptable approach for constructingevaluation metrics and methods for various NLP applications, with a specificfocus on Machine Translation and Summarization. The main objective is toestablish a general framework that can easily support constructing automaticevaluation metrics for a variety of specific NLP tasks and based on a varietyof quality criteria. For a given NLP task (e.g. Machine Translation) and agiven set of established quality criteria, the framework supports learning aset of parameters that result in an "instance" evaluation metric that hasoptimal correlation with the desired quality criteria. Training a new"instance" metric for a different task, or for a different set of qualitycriteria, can be accomplished by a fast training procedure using availabletraining data consisting of system produced outputs, human-quality referenceoutputs for the same source data, and human quality judgments for the systemoutputs.A powerful new innovation of the new framework is its ability to use the setof all overlapping sub-sequences (also known as "skip ngrams") of the twostrings being compared. The process of skip n-gram matching is augmented witha powerful word-to-word alignment algorithm that pre-constrains the set ofskip n-gram matches, while allowing matches between words that aremorphological variants, synonyms or otherwise related. Furthermore, ourframework uses a well-founded parameterized model for establishing the weightor significance that should be assigned to each detected overlappingsubsequence, and can calculate these weights as an integral process during thedetection of the matching skip ngrams. The result is an extremely powerful"metric-producing" framework. Under this framework, the project will produce(instantiate) specific metrics for machine translation, summarization, andother NLP tasks, that are more robust, sensitive, and have high-levels ofcorrelation with human judgments. The project also explores methods forreducing the reliance of our resulting metrics on human judgments. Theresulting framework and task-specific trained metrics will be made publiclyavailable to the NLP research community. The impact of automatic evaluationmethods extends beyond providing a flexible performance measuring mechanismfor NLP tasks. We expect our work to enable customizing evaluation metricsfor specific tasks within a variety of cross-lingual applications, whichshould significantly boost the overall performance of these applications.