U-Statistics for Importance-Weighted Variational Inference
U-Statistics for Importance-Weighted Variational Inference
复制标题
DOI：
10.48550/arxiv.2302.13918
复制
发表时间：
2023-02
期刊：
影响因子：
--
通讯作者：
Javier Burroni
中科院分区：
文献类型：
--
作者：
Javier Burroni
我们提出了使用U-统计量来减少重要加权变分推理中梯度估计的方差。关键的观察是，给定需要$m>1$样本和总共$n>m$样本用于估计的基本梯度估计器，通过对大小为m$的重叠批次的基本估计器进行平均，可以实现比不相交批次更低的方差，正如目前所做的那样。我们使用经典的U-统计量理论来分析方差的减少，并提出了新的近似与理论保证，以确保计算效率。我们发现经验，U-统计方差减少可以导致适度的显着改善推理性能的一系列模型，几乎没有计算成本。
We propose the use of U-statistics to reduce variance for gradient estimation in importance-weighted variational inference. The key observation is that, given a base gradient estimator that requires $m>1$ samples and a total of $n>m$ samples to be used for estimation, lower variance is achieved by averaging the base estimator on overlapping batches of size $m$ than disjoint batches, as currently done. We use classical U-statistic theory to analyze the variance reduction, and propose novel approximations with theoretical guarantees to ensure computational efficiency. We find empirically that U-statistic variance reduction can lead to modest to significant improvements in inference performance on a range of models, with little computational cost.