Subsampled Stochastic Variance-Reduced Gradient Langevin Dynamics
Subsampled Stochastic Variance-Reduced Gradient Langevin Dynamics
复制标题
DOI：
--
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
Difan Zou;Pan Xu;Quanquan Gu
中科院分区：
文献类型：
--
作者：
Difan Zou;Pan Xu;Quanquan Gu
随机梯度朗之万动力学(SVRG-LD)是最近提出的一种通过降低随机梯度的方差来改善随机梯度朗之万动力学(SGLD)性能的方法。在本文中，我们提出了SVRG-LD的一种变体，即SVRG-LD，它用一个次采样的梯度来代替每个历元中的全梯度。我们对SVRG-LD在2-Wasserstein距离上的收敛进行了非渐近分析，并证明了当样本量较大或目标精度要求适中时，SVRG-LD比SVRG-LD具有更低的梯度复杂度。我们的分析直接表明了SVRG-LD具有更快的收敛速度，从而使现有的收敛速度提高了一倍κn，其中κ是对数密度函数的条件数，n是样本量。在合成数据集和真实数据集上的实验验证了我们的理论结果。
Stochastic variance-reduced gradient Langevin dynamics (SVRG-LD) was recently proposed to improve the performance of stochastic gradient Langevin dynamics (SGLD) by reducing the variance of the stochastic gradient. In this paper, we propose a variant of SVRG-LD, namely SVRG-LD, which replaces the full gradient in each epoch with a subsampled one. We provide a nonasymptotic analysis of the convergence of SVRG-LD in 2-Wasserstein distance, and show that SVRG-LD enjoys a lower gradient complexity1 than SVRG-LD, when the sample size is large or the target accuracy requirement is moderate. Our analysis directly implies a sharper convergence rate for SVRG-LD, which improves the existing convergence rate by a factor of κn, where κ is the condition number of the log-density function and n is the sample size. Experiments on both synthetic and real-world datasets validate our theoretical results.