Stein Variational Inference for Discrete Distributions
Stein Variational Inference for Discrete Distributions
复制标题
DOI：
--
复制
发表时间：
2020-03
期刊：
影响因子：
--
通讯作者：
Jun Han;Fan Ding;Xianglong Liu;L. Torresani;Jian Peng;Qiang Liu
中科院分区：
文献类型：
--
作者：
Jun Han;Fan Ding;Xianglong Liu;L. Torresani;Jian Peng;Qiang Liu
基于一致性的近似推理方法，如Stein变分梯度下降（SVGD），为可微连续分布提供了简单而通用的推理引擎。然而，现有形式的SVGD不能直接应用于离散分布。在这项工作中，我们填补了这一空白，提出了一个简单而通用的框架，将离散分布转换为等效的分段连续分布，在此基础上应用无梯度SVGD进行有效的近似推理。实验结果表明，我们的方法优于传统的算法，如吉布斯采样和不连续的哈密顿蒙特卡罗离散图形模型的各种具有挑战性的基准。我们证明，我们的方法为学习二值化神经网络（BNN）的集成提供了一个很有前途的工具，在CIFAR-10数据集上学习二值化AlexNet时，性能优于其他广泛使用的集成方法。此外，这种变换可以直接用于无梯度核化Stein差异，对离散分布进行拟合优度（GOF）检验。我们提出的方法优于现有的GOF测试方法的难处理的离散分布。
Gradient-based approximate inference methods, such as Stein variational gradient descent (SVGD), provide simple and general-purpose inference engines for differentiable continuous distributions. However, existing forms of SVGD cannot be directly applied to discrete distributions. In this work, we fill this gap by proposing a simple yet general framework that transforms discrete distributions to equivalent piecewise continuous distributions, on which the gradient-free SVGD is applied to perform efficient approximate inference. The empirical results show that our method outperforms traditional algorithms such as Gibbs sampling and discontinuous Hamiltonian Monte Carlo on various challenging benchmarks of discrete graphical models. We demonstrate that our method provides a promising tool for learning ensembles of binarized neural network (BNN), outperforming other widely used ensemble methods on learning binarized AlexNet on CIFAR-10 dataset. In addition, such transform can be straightforwardly employed in gradient-free kernelized Stein discrepancy to perform goodness-of-fit (GOF) test on discrete distributions. Our proposed method outperforms existing GOF test methods for intractable discrete distributions.