Constrained Sampling from Language Models via Langevin Dynamics in Embedding Spaces
Constrained Sampling from Language Models via Langevin Dynamics in Embedding Spaces
复制标题
DOI：
10.48550/arxiv.2205.12558
复制
发表时间：
2022
期刊：
影响因子：
--
通讯作者：
Sachin Kumar;Biswajit Paria;Yulia Tsvetkov
中科院分区：
文献类型：
--
作者：
Sachin Kumar;Biswajit Paria;Yulia Tsvetkov
Please try later.
Large pre-trained language models are well-established for their ability to generate text seemingly indistinguishable from humans. In this work, we study the problem of constrained sampling from such language models. That is, generating text that satisfies user-defined constraints. Typical decoding strategies which generate samples left-to-right are not always conducive to imposing such constraints globally. Instead, we propose M U - C O L A —a sampling procedure that combines the log-likelihood of the language model with arbitrary differentiable constraints into a single energy function; and generates samples by initializing the entire output sequence with noise and following a Markov chain defined by Langevin Dynamics using the gradients of this energy. We evaluate our approach on text generation with soft and hard constraints as well as their combinations with competitive results for toxicity avoidance, sentiment control, and keyword guided generation. 1