Variance reduction properties of the reparameterization trick
Variance reduction properties of the reparameterization trick
复制标题
重新参数化技巧的方差减少特性
DOI：
--
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
S. Sisson
中科院分区：
文献类型：
--
作者：
Ming Xu;M. Quiroz;R. Kohn;S. Sisson
重新参数化技巧广泛用于变分推理，因为它比其他方法（如得分函数方法）更准确地估计变分目标的梯度。虽然在文献中有大量的经验证据表明它的成功，但很少有研究探索为什么重新参数化技巧如此有效。我们探讨这一理想化的假设下，变分近似是一个平均场高斯密度和日志的联合密度的模型参数和数据是一个二次函数，取决于变分均值。由此，我们证明了重新参数化梯度估计的边际方差小于得分函数梯度估计。我们将理想化分析的结果应用于现实世界的例子。
The reparameterization trick is widely used in variational inference as it yields more accurate estimates of the gradient of the variational objective than alternative approaches such as the score function method. Although there is overwhelming empirical evidence in the literature showing its success, there is relatively little research exploring why the reparameterization trick is so effective. We explore this under the idealized assumptions that the variational approximation is a mean-field Gaussian density and that the log of the joint density of the model parameters and the data is a quadratic function that depends on the variational mean. From this, we show that the marginal variances of the reparameterization gradient estimator are smaller than those of the score function gradient estimator. We apply the result of our idealized analysis to real-world examples.