Sequence Level Training with Recurrent Neural Networks
Sequence Level Training with Recurrent Neural Networks
复制标题
DOI：
--
复制
发表时间：
2015-11
期刊：
影响因子：
--
通讯作者：
Marc'Aurelio Ranzato;S. Chopra;Michael Auli;Wojciech Zaremba
中科院分区：
文献类型：
--
作者：
Marc'Aurelio Ranzato;S. Chopra;Michael Auli;Wojciech Zaremba
许多自然语言处理应用程序使用语言模型来生成文本。这些模型通常被训练来预测序列中的下一个单词，给定之前的单词和一些上下文，如图像。但是，在测试时，模型预计将从头开始生成整个序列。这种差异使生成变得脆弱，因为错误可能会在过程中沿着累积。我们通过提出一种新的序列级训练算法来解决这个问题，该算法直接优化测试时使用的度量，如BLEU或ROUGE。在三个不同的任务中，我们的方法优于几个强大的贪婪生成基线。当这些基线采用波束搜索时，该方法也具有竞争力，同时速度快了几倍。
Many natural language processing applications use language models to generate text. These models are typically trained to predict the next word in a sequence, given the previous words and some context such as an image. However, at test time the model is expected to generate the entire sequence from scratch. This discrepancy makes generation brittle, as errors may accumulate along the way. We address this issue by proposing a novel sequence level training algorithm that directly optimizes the metric used at test time, such as BLEU or ROUGE. On three different tasks, our approach outperforms several strong baselines for greedy generation. The method is also competitive when these baselines employ beam search, while being several times faster.