On the Blind Spots of Model-Based Evaluation Metrics for Text Generation
On the Blind Spots of Model-Based Evaluation Metrics for Text Generation
复制标题
DOI：
10.48550/arxiv.2212.10020
复制
发表时间：
2022-12
期刊：
影响因子：
--
通讯作者：
Tianxing He;Jingyu Zhang;Tianle Wang;Sachin Kumar;Kyunghyun Cho;James R. Glass;Yulia Tsvetkov
中科院分区：
文献类型：
--
作者：
Tianxing He;Jingyu Zhang;Tianle Wang;Sachin Kumar;Kyunghyun Cho;James R. Glass;Yulia Tsvetkov
在这项工作中，我们探讨了一个有用的，但往往被忽视的方法，用于文本生成评估指标的鲁棒性分析：压力测试与合成数据。基本上，我们设计和综合了各种潜在的错误，并检查它们是否会导致度量分数的相应下降。我们研究了一系列最近提出的基于预训练语言模型的评估指标，用于开放式生成，翻译和摘要的任务。我们的实验揭示了现有指标中有趣的不敏感性，偏见，甚至漏洞。例如，我们发现BERTScore被摘要中的截断错误所迷惑，而MAUVE（建立在GPT-2之上）对代开始或中间的错误不敏感。此外，我们调查了这些盲点背后的原因，并提出了更可靠的文本生成评估的实际解决方案。我们已经在https://github.com/cloudygoose/blindspot_nlg上发布了我们的代码和数据。
In this work, we explore a useful but often neglected methodology for robustness analysis of text generation evaluation metrics: stress tests with synthetic data. Basically, we design and synthesize a wide range of potential errors and check whether they result in a commensurate drop in the metric scores. We examine a range of recently proposed evaluation metrics based on pretrained language models, for the tasks of open-ended generation, translation, and summarization. Our experiments reveal interesting insensitivities, biases, or even loopholes in existing metrics. For example, we find that BERTScore is confused by truncation errors in summarization, and MAUVE (built on top of GPT-2) is insensitive to errors at the beginning or middle of generations. Further, we investigate the reasons behind these blind spots and suggest practical workarounds for a more reliable evaluation of text generation. We have released our code and data at https://github.com/cloudygoose/blindspot_nlg.