Variational Representations and Neural Network Estimation of Rényi Divergences
Variational Representations and Neural Network Estimation of Rényi Divergences
复制标题
DOI：
10.1137/20m1368926
复制
发表时间：
2020-07
期刊：
影响因子：
--
通讯作者：
Jeremiah Birrell;P. Dupuis;M. Katsoulakis;Luc Rey-Bellet;Jie Wang
中科院分区：
文献类型：
--
作者：
Jeremiah Birrell;P. Dupuis;M. Katsoulakis;Luc Rey-Bellet;Jie Wang
我们推导了概率测度 $Q$ 和 $P$ 之间的 R{e}nyi 散度族 $R_\alpha(Q\|P)$ 的新变分公式。我们的结果概括了 Kullback-Leibler 散度的经典 Donsker-Varadhan 变分公式。我们进一步证明这个 R{e}nyi 变分公式在一系列函数空间上成立；这导致了在非常弱的假设下优化器的公式，也是我们开发 R{e}nyi 散度估计器一致性理论的关键。通过将该理论应用于神经网络估计器，我们表明，如果一个神经网络族满足通用逼近性质的几个加强版本之一，则相应的 R{e}nyi 散度估计器是一致的。与基于似然比的方法相比，我们的估计量仅涉及 $Q$ 和 $P$ 下的期望，因此在高维系统中更有效。我们通过在高达 5000 维的系统中进行神经网络估计的几个数值示例来说明这一点。
We derive a new variational formula for the R{e}nyi family of divergences, $R_\alpha(Q\|P)$, between probability measures $Q$ and $P$. Our result generalizes the classical Donsker-Varadhan variational formula for the Kullback-Leibler divergence. We further show that this R{e}nyi variational formula holds over a range of function spaces; this leads to a formula for the optimizer under very weak assumptions and is also key in our development of a consistency theory for R{e}nyi divergence estimators. By applying this theory to neural network estimators, we show that if a neural network family satisfies one of several strengthened versions of the universal approximation property then the corresponding R{e}nyi divergence estimator is consistent. In contrast to likelihood-ratio based methods, our estimators involve only expectations under $Q$ and $P$ and hence are more effective in high dimensional systems. We illustrate this via several numerical examples of neural network estimation in systems of up to 5000 dimensions.