An Improved Analysis of Training Over-parameterized Deep Neural Networks
An Improved Analysis of Training Over-parameterized Deep Neural Networks
复制标题
DOI：
--
复制
发表时间：
2019-06
期刊：
影响因子：
--
通讯作者：
Difan Zou;Quanquan Gu
中科院分区：
文献类型：
--
作者：
Difan Zou;Quanquan Gu
最近的一系列研究表明，具有随机初始化的基于梯度的算法可以收敛到过度参数化（即足够宽）深度神经网络的训练损失的全局最小值。然而，保证全局收敛的神经网络宽度的条件非常严格，通常是训练样本大小$n$中的高次多项式（例如$O(n^{24})$）。在本文中，我们对训练深度神经网络的（随机）梯度下降的全局收敛性进行了改进的分析，与之前的工作相比，在训练样本大小和其他与问题相关的参数方面，它只需要更温和的过参数化条件。我们分析的主要技术贡献包括（a）更严格的梯度下限，导致算法更快收敛，以及（b）更清晰地表征算法的轨迹长度。通过将我们的结果专门化为两层（即单隐藏层）神经网络，它还提供了比先前工作中最著名的结果更温和的过度参数化条件。
A recent line of research has shown that gradient-based algorithms with random initialization can converge to the global minima of the training loss for over-parameterized (i.e., sufficiently wide) deep neural networks. However, the condition on the width of the neural network to ensure the global convergence is very stringent, which is often a high-degree polynomial in the training sample size $n$ (e.g., $O(n^{24})$). In this paper, we provide an improved analysis of the global convergence of (stochastic) gradient descent for training deep neural networks, which only requires a milder over-parameterization condition than previous work in terms of the training sample size and other problem-dependent parameters. The main technical contributions of our analysis include (a) a tighter gradient lower bound that leads to a faster convergence of the algorithm, and (b) a sharper characterization of the trajectory length of the algorithm. By specializing our result to two-layer (i.e., one-hidden-layer) neural networks, it also provides a milder over-parameterization condition than the best-known result in prior work.