BOLT: Fast Energy-based Controlled Text Generation with Tunable Biases
BOLT: Fast Energy-based Controlled Text Generation with Tunable Biases
复制标题
DOI：
10.48550/arxiv.2305.12018
复制
发表时间：
2023-05
期刊：
影响因子：
--
通讯作者：
Xin Liu;Muhammad Khalifa;Lu Wang
中科院分区：
文献类型：
--
作者：
Xin Liu;Muhammad Khalifa;Lu Wang
基于能量的模型(EBM)由于其对各种约束的高度适用性而在受控文本生成方面获得了广泛的应用。然而，从EBMS进行采样并不容易，因为它通常需要大量迭代才能收敛到可信的文本，这会减慢解码过程，并使其不太适用于现实世界的应用程序。在这项工作中，我们提出了Bolt，它依靠可调的偏差来直接调整语言模型的输出日志。与以前的工作不同，Bolt保持了生成器的自回归性质，以断言对令牌条件依赖和整体流畅性的强大控制，因此收敛速度更快。与使用软约束(例如，情感控制)和硬约束(例如，关键字引导的主题控制)的受控生成任务的最新技术相比，Bolt的效率和流畅性都有了显著的提高。根据人类评委的数据，在情绪控制方面，Bolt的速度是竞争基线的7倍，在74.4%的评估样本中更流畅。
Energy-based models (EBMs) have gained popularity for controlled text generation due to their high applicability to a wide range of constraints. However, sampling from EBMs is non-trivial, as it often requires a large number of iterations to converge to plausible text, which slows down the decoding process and makes it less practical for real-world applications. In this work, we propose BOLT, which relies on tunable biases to directly adjust the language model’s output logits. Unlike prior work, BOLT maintains the generator’s autoregressive nature to assert a strong control on token-wise conditional dependencies and overall fluency, and thus converges faster. When compared with state-of-the-arts on controlled generation tasks using both soft constraints (e.g., sentiment control) and hard constraints (e.g., keyword-guided topic control), BOLT demonstrates significantly improved efficiency and fluency. On sentiment control, BOLT is 7x faster than competitive baselines, and more fluent in 74.4% of the evaluation samples according to human judges.