Attribute Alignment: Controlling Text Generation from Pre-trained Language Models
Attribute Alignment: Controlling Text Generation from Pre-trained Language Models
复制标题
DOI：
10.18653/v1/2021.findings-emnlp.194
复制
发表时间：
2021-03
期刊：
影响因子：
--
通讯作者：
Dian Yu;Kenji Sagae;Zhou Yu
中科院分区：
文献类型：
--
作者：
Dian Yu;Kenji Sagae;Zhou Yu
大型语言模型受益于大量未标记文本的训练，这使它们具有越来越流畅和多样化的生成能力。然而，使用这些模型来生成考虑目标属性（如情感极性或特定主题）的文本仍然是一个挑战。我们提出了一个简单而灵活的方法来控制文本生成对齐解开属性表示。与最近训练一个正则表达式来扰乱属性的标记级别分布的努力相反，我们使用相同的数据来学习对齐函数，以指导预训练的非控制语言模型生成具有目标属性的文本，而不改变原始语言模型参数。我们评估了我们的方法的情绪和主题控制的生成，并显示出比以前的方法大的性能增益，同时保持流畅性和多样性。
Large language models benefit from training with a large amount of unlabeled text, which gives them increasingly fluent and diverse generation capabilities. However, using these models for text generation that takes into account target attributes, such as sentiment polarity or specific topics, remains a challenge. We propose a simple and flexible method for controlling text generation by aligning disentangled attribute representations. In contrast to recent efforts on training a discriminator to perturb the token level distribution for an attribute, we use the same data to learn an alignment function to guide the pre-trained, non-controlled language model to generate texts with the target attribute without changing the original language model parameters. We evaluate our method on sentiment- and topic-controlled generation, and show large performance gains over previous methods while retaining fluency and diversity.