Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networks
Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networks
复制标题
DOI：
--
复制
发表时间：
2021-09
期刊：
影响因子：
--
通讯作者：
Hao Liu;Minshuo Chen;T. Zhao;Wenjing Liao
中科院分区：
文献类型：
--
作者：
Hao Liu;Minshuo Chen;T. Zhao;Wenjing Liao
现有的大多数关于深度神经网络的统计理论都存在样本复杂性受到数据维度的影响，因此不能很好地解释高维数据上深度学习的经验成功。为了弥补这一差距，我们建议利用现实世界数据集的低维几何结构。我们从函数逼近和二分类统计估计两个方面建立了卷积剩余网络(ConvResNet)的理论保证。具体地说，假设数据位于等距嵌入到流形中的$d$维流形上，我们证明了如果选择适当的网络结构，对应网可以(1)以任意精度逼近流形上的Besov函数，(2)通过最小化经验Logistic风险来学习分类器，这给出了$n^{-\FRAC{S}{2s+2(S)}}$的超额风险，其中$S$是光滑性参数。这意味着样本复杂性取决于内在维度$d$，而不是数据维度$D$。我们的结果表明，ConvResNets对数据集的低维结构具有很好的适应性。
Most of existing statistical theories on deep neural networks have sample complexities cursed by the data dimension and therefore cannot well explain the empirical success of deep learning on high-dimensional data. To bridge this gap, we propose to exploit low-dimensional geometric structures of the real world data sets. We establish theoretical guarantees of convolutional residual networks (ConvResNet) in terms of function approximation and statistical estimation for binary classification. Specifically, given the data lying on a $d$-dimensional manifold isometrically embedded in $\mathbb{R}^D$, we prove that if the network architecture is properly chosen, ConvResNets can (1) approximate Besov functions on manifolds with arbitrary accuracy, and (2) learn a classifier by minimizing the empirical logistic risk, which gives an excess risk in the order of $n^{-\frac{s}{2s+2(s\vee d)}}$, where $s$ is a smoothness parameter. This implies that the sample complexity depends on the intrinsic dimension $d$, instead of the data dimension $D$. Our results demonstrate that ConvResNets are adaptive to low-dimensional structures of data sets.