SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control
SSD-LM: Semi-autoregressive Simplex-based Diffusion Language Model for Text Generation and Modular Control
复制标题
DOI：
10.48550/arxiv.2210.17432
复制
发表时间：
2022-10
期刊：
影响因子：
--
通讯作者：
Xiaochuang Han;Sachin Kumar;Yulia Tsvetkov
中科院分区：
文献类型：
--
作者：
Xiaochuang Han;Sachin Kumar;Yulia Tsvetkov
尽管扩散模型在连续值领域（如图像）取得了越来越大的成功，但在离散领域（如文本）的类似努力尚未与自回归语言模型的性能相匹配。在这项工作中，我们提出了基于扩散的语言模型ssd - lm，其中有两个关键的设计选择。首先，SSD-LM是半自回归的，迭代地生成文本块，在解码时允许灵活的输出长度，同时支持本地双向上下文更新。其次，它是基于简单的，在自然词汇空间而不是学习的潜在空间上执行扩散，允许我们使用现成的分类器结合分类器引导和模块化控制，而不需要任何自适应。我们在无约束的文本生成基准上评估了SSD-LM，并表明它在标准质量和多样性指标上匹配或优于强大的自回归GPT-2模型，同时大大优于基于扩散的基线。在受控文本生成方面，SSD-LM在模块化方面具有额外的优势，也优于竞争基准。
Despite the growing success of diffusion models in continuous-valued domains (e.g., images), similar efforts for discrete domains such as text have yet to match the performance of autoregressive language models. In this work, we present SSD-LM—a diffusion-based language model with two key design choices. First, SSD-LM is semi-autoregressive, iteratively generating blocks of text, allowing for flexible output length at decoding time while enabling local bidirectional context updates. Second, it is simplex-based, performing diffusion on the natural vocabulary space rather than a learned latent space, allowing us to incorporate classifier guidance and modular control using off-the-shelf classifiers without any adaptation. We evaluate SSD-LM on unconstrained text generation benchmarks, and show that it matches or outperforms strong autoregressive GPT-2 models across standard quality and diversity metrics, while vastly outperforming diffusion-based baselines. On controlled text generation, SSD-LM also outperforms competitive baselines, with an extra advantage in modularity.