A Framework for Learning High Accuracy Evaluation Metrics for NLP Applications

点击翻译按钮获取中文摘要

This project investigates a new dynamic, adaptable approach for constructingevaluation metrics and methods for various NLP applications, with a specificfocus on Machine Translation and Summarization. The main objective is toestablish a general framework that can easily support constructing automaticevaluation metrics for a variety of specific NLP tasks and based on a varietyof quality criteria. For a given NLP task (e.g. Machine Translation) and agiven set of established quality criteria, the framework supports learning aset of parameters that result in an "instance" evaluation metric that hasoptimal correlation with the desired quality criteria. Training a new"instance" metric for a different task, or for a different set of qualitycriteria, can be accomplished by a fast training procedure using availabletraining data consisting of system produced outputs, human-quality referenceoutputs for the same source data, and human quality judgments for the systemoutputs.A powerful new innovation of the new framework is its ability to use the setof all overlapping sub-sequences (also known as "skip ngrams") of the twostrings being compared. The process of skip n-gram matching is augmented witha powerful word-to-word alignment algorithm that pre-constrains the set ofskip n-gram matches, while allowing matches between words that aremorphological variants, synonyms or otherwise related. Furthermore, ourframework uses a well-founded parameterized model for establishing the weightor significance that should be assigned to each detected overlappingsubsequence, and can calculate these weights as an integral process during thedetection of the matching skip ngrams. The result is an extremely powerful"metric-producing" framework. Under this framework, the project will produce(instantiate) specific metrics for machine translation, summarization, andother NLP tasks, that are more robust, sensitive, and have high-levels ofcorrelation with human judgments. The project also explores methods forreducing the reliance of our resulting metrics on human judgments. Theresulting framework and task-specific trained metrics will be made publiclyavailable to the NLP research community. The impact of automatic evaluationmethods extends beyond providing a flexible performance measuring mechanismfor NLP tasks. We expect our work to enable customizing evaluation metricsfor specific tasks within a variety of cross-lingual applications, whichshould significantly boost the overall performance of these applications.