Gradient descent optimizes over-parameterized deep ReLU networks
Gradient descent optimizes over-parameterized deep ReLU networks
复制标题
DOI：
10.1007/s10994-019-05839-6
复制
发表时间：
2019-10-23
期刊：
影响因子：
7.5
通讯作者：
Gu, Quanquan
中科院分区：
文献类型：
--
作者：
Zou, Difan;Cao, Yuan;Gu, Quanquan
我们研究了使用梯度下降训练具有整流线性单元（ReLU）激活函数和交叉熵损失函数的深度全连接神经网络进行二进制分类的问题。我们表明，在对训练数据进行一定假设的情况下，通过适当的随机权重初始化，梯度下降可以找到过度参数化的深度ReLU网络的训练损失的全局最小值。我们证明的关键思想是高斯随机初始化，然后梯度下降产生一系列迭代，这些迭代停留在以初始权重为中心的小扰动区域内，其中深度ReLU网络的训练损失函数具有良好的局部曲率属性，确保梯度下降的全局收敛性。我们证明技术的核心是（1）对训练数据的温和假设;（2）对梯度下降的轨迹长度的尖锐分析;（3）对扰动区域大小的更精细表征。与并行工作（Allen-Zhu et al. in A convergence theory for deep learning via over-parameterization，2018 a; Du et al. in Gradient descent finds global minima of deep neural networks，2018 a）相比，沿着这条线，我们的结果依赖于神经网络宽度上更温和的过参数化条件，并且在训练深度神经网络时具有更快的梯度下降全局收敛速度。
We study the problem of training deep fully connected neural networks with Rectified Linear Unit (ReLU) activation function and cross entropy loss function for binary classification using gradient descent. We show that with proper random weight initialization, gradient descent can find the global minima of the training loss for an over-parameterized deep ReLU network, under certain assumption on the training data. The key idea of our proof is that Gaussian random initialization followed by gradient descent produces a sequence of iterates that stay inside a small perturbation region centered at the initial weights, in which the training loss function of the deep ReLU networks enjoys nice local curvature properties that ensure the global convergence of gradient descent. At the core of our proof technique is (1) a milder assumption on the training data; (2) a sharp analysis of the trajectory length for gradient descent; and (3) a finer characterization of the size of the perturbation region. Compared with the concurrent work (Allen-Zhu et al. in A convergence theory for deep learning via over-parameterization, 2018a; Du et al. in Gradient descent finds global minima of deep neural networks, 2018a) along this line, our result relies on milder over-parameterization condition on the neural network width, and enjoys faster global convergence rate of gradient descent for training deep neural networks.