A Geometric Approach of Gradient Descent Algorithms in Linear Neural Networks
A Geometric Approach of Gradient Descent Algorithms in Linear Neural Networks
复制标题
线性神经网络中梯度下降算法的几何方法
DOI：
--
复制
发表时间：
2018
期刊：
影响因子：
--
通讯作者：
Romain Couillet
中科院分区：
文献类型：
--
作者：
Y. Chitour;Zhenyu Liao;Romain Couillet
在本文中，我们提出了一个几何框架来分析线性神经网络的梯度下降轨迹的收敛特性。我们将线性神经网络的一个著名的经验观察转化为一个猜想，我们称之为过拟合猜想，即对于几乎所有的训练数据和初始条件，相应的梯度下降系统的轨迹收敛到全局最小值。这意味着对于任意数量的隐藏层的线性神经网络，通过香草梯度下降算法实现的解决方案等价于最小二乘估计。建立在一个关键的不变性属性诱导的网络结构，我们首先建立收敛的梯度下降轨迹的平方损失函数的临界点的情况下，任意深度的线性网络。我们的第二个结果是在单隐藏层线性网络的情况下证明了过拟合猜想，其中参数基于正常双曲性的概念并且在训练数据的一般属性下（即，对于几乎所有的训练数据保持不变）。
In this paper, we propose a geometric framework to analyze the convergence properties of gradient descent trajectories in the context of linear neural networks. We translate a well-known empirical observation of linear neural nets into a conjecture that we call the emph{overfitting conjecture} which states that, for almost all training data and initial conditions, the trajectory of the corresponding gradient descent system converges to a global minimum. This would imply that the solution achieved by vanilla gradient descent algorithms is equivalent to that of the least-squares estimation, for linear neural networks of an arbitrary number of hidden layers. Built upon a key invariance property induced by the network structure, we first establish convergence of gradient descent trajectories to critical points of the square loss function in the case of linear networks of arbitrary depth. Our second result is the proof of the emph{overfitting conjecture} in the case of single-hidden-layer linear networks with an argument based on the notion of normal hyperbolicity and under a generic property on the training data (i.e., holding for almost all training data).