PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation
PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation
复制标题
DOI：
10.18653/v1/2020.emnlp-main.57
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
Xinyu Hua;Lu Wang
中科院分区：
文献类型：
--
作者：
Xinyu Hua;Lu Wang
经过预先训练的变形金刚在生成长而流畅的文本方面取得了令人印象深刻的突破，但他们的输出往往是杂乱无章的，没有连贯的内容安排。在这项工作中，我们提出了一种新的内容控制文本生成框架Pair，该框架具有规划和迭代求精功能，该框架建立在一个大型模型BART的基础上。我们首先采用ERT模型来自动构建内容计划，由关键短语分配及其对应的句子级位置组成。在不改变其结构的情况下，采用BART模型进行生成。然后，我们提出了一种精化算法，在序列到序列的框架内逐步提高生成质量。自动度量评估显示，添加规划在三个不同的域上一致地提高了生成质量，平均提高了20个BLEU点和12个流星点。此外，人类评委对我们的系统输出的评价比没有规划的比较更相关、更连贯。
Pre-trained Transformers have enabled impressive breakthroughs in generating long and fluent text, yet their outputs are often "rambling" without coherently arranged content. In this work, we present a novel content-controlled text generation framework, PAIR, with planning and iterative refinement, which is built upon a large model, BART. We first adapt the BERT model to automatically construct the content plans, consisting of keyphrase assignments and their corresponding sentence-level positions. The BART model is employed for generation without modifying its structure. We then propose a refinement algorithm to gradually enhance the generation quality within the sequence-to-sequence framework. Evaluation with automatic metrics shows that adding planning consistently improves the generation quality on three distinct domains, with an average of 20 BLEU points and 12 METEOR points improvements. In addition, human judges rate our system outputs to be more relevant and coherent than comparisons without planning.