Benefits of Overparameterized Convolutional Residual Networks: Function Approximation under Smoothness Constraint
Benefits of Overparameterized Convolutional Residual Networks: Function Approximation under Smoothness Constraint
复制标题
DOI：
10.48550/arxiv.2206.04569
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
Hao Liu;Minshuo Chen;Siawpeng Er;Wenjing Liao;Tong-Mei Zhang;Tuo Zhao
中科院分区：
文献类型：
--
作者：
Hao Liu;Minshuo Chen;Siawpeng Er;Wenjing Liao;Tong-Mei Zhang;Tuo Zhao
过参数化神经网络对复杂数据具有很强的表示能力，更重要的是产生足够平滑的输出，这对它们的泛化和鲁棒性至关重要。大多数现有的函数逼近理论表明，有足够多的参数，神经网络可以很好地近似某些类别的函数的函数值。然而，神经网络本身可能是高度非光滑的。为了弥补这一差距，我们以卷积残差网络（ConvResNets）为例，证明大型ConvResNets不仅可以在函数值方面近似目标函数，而且还表现出足够的一阶平滑性。此外，我们扩展我们的理论，以逼近低维流形上支持的函数。我们的理论部分证明了在实践中使用深度和广度网络的好处。对抗性鲁棒图像分类的数值实验提供了支持我们的理论。
Overparameterized neural networks enjoy great representation power on complex data, and more importantly yield sufficiently smooth output, which is crucial to their generalization and robustness. Most existing function approximation theories suggest that with sufficiently many parameters, neural networks can well approximate certain classes of functions in terms of the function value. The neural network themselves, however, can be highly nonsmooth. To bridge this gap, we take convolutional residual networks (ConvResNets) as an example, and prove that large ConvResNets can not only approximate a target function in terms of function value, but also exhibit sufficient first-order smoothness. Moreover, we extend our theory to approximating functions supported on a low-dimensional manifold. Our theory partially justifies the benefits of using deep and wide networks in practice. Numerical experiments on adversarial robust image classification are provided to support our theory.