Toward Moderate Overparameterization: Global Convergence Guarantees for Training Shallow Neural Networks
Toward Moderate Overparameterization: Global Convergence Guarantees for Training Shallow Neural Networks
复制标题
DOI：
10.1109/jsait.2020.2991332
复制
发表时间：
2019-02
期刊：
影响因子：
--
通讯作者：
Samet Oymak;M. Soltanolkotabi
中科院分区：
文献类型：
--
作者：
Samet Oymak;M. Soltanolkotabi
许多现代神经网络架构都是在过参数化机制中训练的，其中模型的参数超过了训练数据集的大小。充分过参数化的神经网络架构原则上有能力拟合任何标签集，包括随机噪声。然而，考虑到训练环境的高度非凸性，尚不清楚一阶方法需要什么水平和种类的过参数化才能收敛到完美插值任何标签的全局最优值。最近的一些理论工作表明，对于非常宽的神经网络，其中隐藏单元的数量在训练数据的大小中是多项式大的，从随机初始化开始的梯度下降确实收敛到全局最优值。然而，在实践中，更温和的过度参数化水平似乎是足够的，在许多情况下，过度参数化模型似乎完美地插值的训练数据，只要参数的数量超过一个常数因子的训练数据的大小。因此，现有的理论文献和实际实验之间存在着巨大的差距。在本文中，我们将采取措施缩小这一差距。聚焦于浅层神经网络和平滑激活，我们证明了（随机）梯度下降在随机初始化时，只要网络参数数量的平方根超过训练数据的大小，就会以几何速度收敛到附近的全局最优值。我们的结果还受益于快速的收敛速度，并继续适用于不可微的激活，如整流线性单元（ReLU）。
Many modern neural network architectures are trained in an overparameterized regime where the parameters of the model exceed the size of the training dataset. Sufficiently overparameterized neural network architectures in principle have the capacity to fit any set of labels including random noise. However, given the highly nonconvex nature of the training landscape it is not clear what level and kind of overparameterization is required for first order methods to converge to a global optima that perfectly interpolate any labels. A number of recent theoretical works have shown that for very wide neural networks where the number of hidden units is polynomially large in the size of the training data gradient descent starting from a random initialization does indeed converge to a global optima. However, in practice much more moderate levels of overparameterization seems to be sufficient and in many cases overparameterized models seem to perfectly interpolate the training data as soon as the number of parameters exceed the size of the training data by a constant factor. Thus there is a huge gap between the existing theoretical literature and practical experiments. In this paper we take a step towards closing this gap. Focusing on shallow neural nets and smooth activations, we show that (stochastic) gradient descent when initialized at random converges at a geometric rate to a nearby global optima as soon as the square-root of the number of network parameters exceeds the size of the training data. Our results also benefit from a fast convergence rate and continue to hold for non-differentiable activations such as Rectified Linear Units (ReLUs).