Latent Diffusion Energy-Based Model for Interpretable Text Modeling
Latent Diffusion Energy-Based Model for Interpretable Text Modeling
复制标题
DOI：
10.48550/arxiv.2206.05895
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
Peiyu Yu;Sirui Xie;Xiaojian Ma;Baoxiong Jia;Bo Pang;Ruigi Gao;Yixin Zhu;Song-Chun Zhu;Y. Wu
中科院分区：
文献类型：
--
作者：
Peiyu Yu;Sirui Xie;Xiaojian Ma;Baoxiong Jia;Bo Pang;Ruigi Gao;Yixin Zhu;Song-Chun Zhu;Y. Wu
基于潜在空间能量的模型（EBMs），也被称为基于能量的先验，在生成模型中引起了越来越多的兴趣。由于潜在空间在表述上的灵活性和强大的建模能力，近年来在此基础上建立的作品对文本建模的可解释性进行了有趣的尝试。然而，潜在空间EBMs也继承了数据空间EBMs的一些缺陷；实践中退化的MCMC采样质量会导致生成质量差和训练不稳定，特别是对于具有复杂潜在结构的数据。受最近利用扩散恢复似然学习作为采样问题解决方案的努力的启发，我们在变分学习框架中引入了扩散模型和潜在空间EBMs之间的新型共生关系，称为基于潜在扩散能量的模型。结合信息瓶颈，提出了一种基于几何聚类的正则化方法，进一步提高了学习到的潜在空间质量。在几个具有挑战性的任务上的实验表明，我们的模型在可解释文本建模方面的性能优于强模型。
Latent space Energy-Based Models (EBMs), also known as energy-based priors, have drawn growing interests in generative modeling. Fueled by its flexibility in the formulation and strong modeling power of the latent space, recent works built upon it have made interesting attempts aiming at the interpretability of text modeling. However, latent space EBMs also inherit some flaws from EBMs in data space; the degenerate MCMC sampling quality in practice can lead to poor generation quality and instability in training, especially on data with complex latent structures. Inspired by the recent efforts that leverage diffusion recovery likelihood learning as a cure for the sampling issue, we introduce a novel symbiosis between the diffusion models and latent space EBMs in a variational learning framework, coined as the latent diffusion energy-based model. We develop a geometric clustering-based regularization jointly with the information bottleneck to further improve the quality of the learned latent space. Experiments on several challenging tasks demonstrate the superior performance of our model on interpretable text modeling over strong counterparts.