A global convergence theory for deep ReLU implicit networks via over-parameterization
A global convergence theory for deep ReLU implicit networks via over-parameterization
复制标题
DOI：
--
复制
发表时间：
2021-10
期刊：
影响因子：
--
通讯作者：
Tianxiang Gao;Hailiang Liu;Jia Liu;Hridesh Rajan;Hongyang Gao
中科院分区：
文献类型：
--
作者：
Tianxiang Gao;Hailiang Liu;Jia Liu;Hridesh Rajan;Hongyang Gao
由于隐式深度学习推广了许多常用神经网络结构的递归预测规则，近年来受到越来越多的关注。它的预测规则是基于一个平衡方程的解来隐式给出的。虽然最近的一系列实证研究已经证明了隐式神经网络的优越性能，但对隐式神经网络的理论理解是有限的。一般来说，在训练过程中，平衡方程可能不是适定的。因此，不能保证香草(随机)梯度下降(SGD)训练的非线性隐式神经网络能够收敛。本文通过分析修正线性单元(RELU)激活的隐式神经网络的梯度流来填补这一空白。对于具有RELU激活和$n$训练样本的$m$宽度的隐式神经网络，我们证明了如果隐式神经网络是{过参数的}，则随机初始化的梯度下降以线性速度收敛于平方损失函数的全局最小值。值得注意的是，与已有的关于(S)GD在有限层超参数神经网络上收敛的工作不同，我们的收敛结果适用于隐式神经网络，这里的层数为{无限}。
Implicit deep learning has received increasing attention recently due to the fact that it generalizes the recursive prediction rules of many commonly used neural network architectures. Its prediction rule is provided implicitly based on the solution of an equilibrium equation. Although a line of recent empirical studies has demonstrated its superior performances, the theoretical understanding of implicit neural networks is limited. In general, the equilibrium equation may not be well-posed during the training. As a result, there is no guarantee that a vanilla (stochastic) gradient descent (SGD) training nonlinear implicit neural networks can converge. This paper fills the gap by analyzing the gradient flow of Rectified Linear Unit (ReLU) activated implicit neural networks. For an $m$-width implicit neural network with ReLU activation and $n$ training samples, we show that a randomly initialized gradient descent converges to a global minimum at a linear rate for the square loss function if the implicit neural network is \textit{over-parameterized}. It is worth noting that, unlike existing works on the convergence of (S)GD on finite-layer over-parameterized neural networks, our convergence results hold for implicit neural networks, where the number of layers is \textit{infinite}.