Non-asymptotic Performance Guarantees for Neural Estimation of f-Divergences
Non-asymptotic Performance Guarantees for Neural Estimation of f-Divergences
复制标题
DOI：
--
复制
发表时间：
2021-03
期刊：
影响因子：
--
通讯作者：
Sreejith Sreekumar;Zhengxin Zhang;Ziv Goldfeld
中科院分区：
文献类型：
--
作者：
Sreejith Sreekumar;Zhengxin Zhang;Ziv Goldfeld
统计距离是机器学习和统计的核心，它量化了概率分布之间的差异。一种从数据中估计这种距离的现代方法依赖于神经网络(NN)对变分形式进行参数化并对其进行优化。这些估计器在实践中得到了大量的应用，但相应的性能保证是片面的，需要进一步探索。特别是，在涉及的两个误差来源之间似乎有一个基本的权衡：近似和估计。前者需要神经网络类的丰富性和表现力，而后者则依赖于控制复杂性。本文通过非渐近误差界的方法对这一权衡进行了探讨，重点讨论了三种流行的选择--Kullback-Leibler散度、卡方散度和平方Hellinger距离。我们的分析依赖于经验过程理论中的非渐近函数逼近定理和工具。文中还给出了验证理论的数值结果。
Statistical distances (SDs), which quantify the dissimilarity between probability distributions, are central to machine learning and statistics. A modern method for estimating such distances from data relies on parametrizing a variational form by a neural network (NN) and optimizing it. These estimators are abundantly used in practice, but corresponding performance guarantees are partial and call for further exploration. In particular, there seems to be a fundamental tradeoff between the two sources of error involved: approximation and estimation. While the former needs the NN class to be rich and expressive, the latter relies on controlling complexity. This paper explores this tradeoff by means of non-asymptotic error bounds, focusing on three popular choices of SDs -- Kullback-Leibler divergence, chi-squared divergence, and squared Hellinger distance. Our analysis relies on non-asymptotic function approximation theorems and tools from empirical process theory. Numerical results validating the theory are also provided.