Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
复制标题
DOI：
--
复制
发表时间：
2019-05
期刊：
影响因子：
--
通讯作者：
Yuan Cao;Quanquan Gu
中科院分区：
文献类型：
--
作者：
Yuan Cao;Quanquan Gu
研究了深度神经网络(DNN)在网络宽度(即每层隐含节点数)远大于训练数据点数目的过参数环境下的训练和泛化问题。我们证明，用随机梯度下降(SGD)和随机初始化训练的足够宽的REU网络预期的$0$-$1$损失可以被初始化时由网络梯度引起的随机特征模型的训练损失所限制，我们称之为神经切线随机特征(NTRF)模型。对于可以用NTRF模型以足够小的误差进行分类的数据分布，我们的结果得到了一个与网络宽度无关的数量级的推广误差界。我们的结果比许多现有的过参数神经网络的泛化误差界更具一般性和敏捷性。此外，我们建立了推广误差界与最近工作中提出的神经切核(NTK)之间的强联系。
We study the training and generalization of deep neural networks (DNNs) in the over-parameterized regime, where the network width (i.e., number of hidden nodes per layer) is much larger than the number of training data points. We show that, the expected $0$-$1$ loss of a wide enough ReLU network trained with stochastic gradient descent (SGD) and random initialization can be bounded by the training loss of a random feature model induced by the network gradient at initialization, which we call a neural tangent random feature (NTRF) model. For data distributions that can be classified by NTRF model with sufficiently small error, our result yields a generalization error bound in the order of $\tilde{\mathcal{O}}(n^{-1/2})$ that is independent of the network width. Our result is more general and sharper than many existing generalization error bounds for over-parameterized neural networks. In addition, we establish a strong connection between our generalization error bound and the neural tangent kernel (NTK) proposed in recent work.