A Framework for Learning High Accuracy Evaluation Metrics for NLP Applications

该项目研究了一种新的动态、适应性强的方法，用于构建各种自然语言处理应用的评估指标和方法，特别关注机器翻译和摘要。主要目标是建立一个通用框架，该框架可以很容易地支持为各种特定的NLP任务构建基于各种质量标准的自动评估度量。对于给定的NLP任务（例如机器翻译）和给定的一组已建立的质量标准，该框架支持学习一组参数，这些参数产生与所需质量标准具有最佳相关性的“实例”评估度量。为不同的任务或不同的质量标准集训练一个新的“实例”度量，可以通过使用可用的训练数据的快速训练程序来完成，这些数据包括系统产生的输出、相同源数据的人类质量参考输出和系统输出的人类质量判断。新框架的一个强大的新创新是它能够使用被比较的两个字符串的所有重叠子序列的集合（也称为“跳过ngrams”）。skip n-gram匹配过程通过强大的单词对单词对齐算法得到增强，该算法预先限制了skip n-gram匹配集，同时允许词形变体、同义词或其他相关单词之间的匹配。此外，我们的框架使用了一个建立良好的参数化模型来建立应该分配给每个检测到的重叠子序列的权重重要性，并且可以在检测匹配跳跃图的过程中作为一个整体过程来计算这些权重。其结果是一个极其强大的“度量生成”框架。在此框架下，该项目将为机器翻译、摘要和其他NLP任务生成（实例化）特定的度量，这些任务更健壮、更敏感，并且与人类判断具有高度的相关性。该项目还探索了减少对人类判断的依赖的方法。由此产生的框架和特定任务的训练指标将公开提供给NLP研究界。自动评估方法的影响超出了为NLP任务提供灵活的性能测量机制。我们希望我们的工作能够为各种跨语言应用程序中的特定任务定制评估指标，这将显著提高这些应用程序的整体性能。

This project investigates a new dynamic, adaptable approach for constructingevaluation metrics and methods for various NLP applications, with a specificfocus on Machine Translation and Summarization. The main objective is toestablish a general framework that can easily support constructing automaticevaluation metrics for a variety of specific NLP tasks and based on a varietyof quality criteria. For a given NLP task (e.g. Machine Translation) and agiven set of established quality criteria, the framework supports learning aset of parameters that result in an "instance" evaluation metric that hasoptimal correlation with the desired quality criteria. Training a new"instance" metric for a different task, or for a different set of qualitycriteria, can be accomplished by a fast training procedure using availabletraining data consisting of system produced outputs, human-quality referenceoutputs for the same source data, and human quality judgments for the systemoutputs.A powerful new innovation of the new framework is its ability to use the setof all overlapping sub-sequences (also known as "skip ngrams") of the twostrings being compared. The process of skip n-gram matching is augmented witha powerful word-to-word alignment algorithm that pre-constrains the set ofskip n-gram matches, while allowing matches between words that aremorphological variants, synonyms or otherwise related. Furthermore, ourframework uses a well-founded parameterized model for establishing the weightor significance that should be assigned to each detected overlappingsubsequence, and can calculate these weights as an integral process during thedetection of the matching skip ngrams. The result is an extremely powerful"metric-producing" framework. Under this framework, the project will produce(instantiate) specific metrics for machine translation, summarization, andother NLP tasks, that are more robust, sensitive, and have high-levels ofcorrelation with human judgments. The project also explores methods forreducing the reliance of our resulting metrics on human judgments. Theresulting framework and task-specific trained metrics will be made publiclyavailable to the NLP research community. The impact of automatic evaluationmethods extends beyond providing a flexible performance measuring mechanismfor NLP tasks. We expect our work to enable customizing evaluation metricsfor specific tasks within a variety of cross-lingual applications, whichshould significantly boost the overall performance of these applications.