Sharp Bounds on the Approximation Rates, Metric Entropy, and n-Widths of Shallow Neural Networks
Sharp Bounds on the Approximation Rates, Metric Entropy, and n-Widths of Shallow Neural Networks
复制标题
DOI：
10.1007/s10208-022-09595-3
复制
发表时间：
2021-01
影响因子：
3
通讯作者：
Jonathan W. Siegel;Jinchao Xu
中科院分区：
文献类型：
--
作者：
Jonathan W. Siegel;Jinchao Xu
在本文中，我们研究了与具有各种激活函数的浅层神经网络相对应的变化空间的近似属性。我们引入了两个主要工具来估计这些空间的度量熵、近似率和 n 宽度。首先，我们引入平滑参数化字典的概念，并给出非线性逼近率、度量熵和绝对凸包的 n 宽度的上限。上限取决于参数化的平滑度。该结果应用于与浅层神经网络相对应的岭函数字典，并且它们在许多情况下改进了现有结果。接下来，我们提供了一种对包含某些岭函数类别的变分空间的度量熵和 n 宽度进行下限的方法。该结果给出了对应于具有一系列重要激活函数（包括 ReLU 激活函数和具有有界变化的 sigmoidal 激活函数）的神经网络的变化空间的近似率、度量熵和 n 宽度的急剧下界。
In this article, we study approximation properties of the variation spaces corresponding to shallow neural networks with a variety of activation functions. We introduce two main tools for estimating the metric entropy, approximation rates, andn-widths of these spaces. First, we introduce the notion of a smoothly parameterized dictionary and give upper bounds on the nonlinear approximation rates, metric entropy, andn-widths of their absolute convex hull. The upper bounds depend upon the order of smoothness of the parameterization. This result is applied to dictionaries of ridge functions corresponding to shallow neural networks, and they improve upon existing results in many cases. Next, we provide a method for lower bounding the metric entropy andn-widths of variation spaces which contain certain classes of ridge functions. This result gives sharp lower bounds on the-approximation rates, metric entropy, andn-widths for variation spaces corresponding to neural networks with a range of important activation functions, including ReLUactivation functions and sigmoidal activation functions with bounded variation.