A unified view of likelihood ratio and reparameterization gradients
A unified view of likelihood ratio and reparameterization gradients
复制标题
DOI：
--
复制
发表时间：
2021-05
期刊：
影响因子：
--
通讯作者：
Paavo Parmas;Masashi Sugiyama
中科院分区：
文献类型：
--
作者：
Paavo Parmas;Masashi Sugiyama
在整个机器学习和强化学习中，重新参数化(RP)和似然比(LR)梯度估计器被用来估计期望的梯度；然而，它们通常被解释为简单的数学技巧，没有深入了解它们的性质。我们用第一原理的方法解释了LR和RP是跟踪概率质量运动的替代方法，并且两者通过散度定理联系在一起。此外，我们证明了所有可能结合LR和Rp的估计量的空间可以完全由一个流场$u(X)$和一个重要抽样分布$q(X)$来参数化。我们证明了在我们的特征空间之外不可能存在这种类型的单样本估计，从而阐明了我们应该在哪里寻找更好的蒙特卡罗梯度估计。
Reparameterization (RP) and likelihood ratio (LR) gradient estimators are used to estimate gradients of expectations throughout machine learning and reinforcement learning; however, they are usually explained as simple mathematical tricks, with no insight into their nature. We use a first principles approach to explain that LR and RP are alternative methods of keeping track of the movement of probability mass, and the two are connected via the divergence theorem. Moreover, we show that the space of all possible estimators combining LR and RP can be completely parameterized by a flow field $u(x)$ and an importance sampling distribution $q(x)$. We prove that there cannot exist a single-sample estimator of this type outside our characterized space, thus, clarifying where we should be searching for better Monte Carlo gradient estimators.