Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
复制标题
深度学习中 Hessian 矩阵的特征值：奇点及超越
DOI：
--
复制
发表时间：
2016
期刊：
影响因子：
--
通讯作者：
Yann LeCun
中科院分区：
文献类型：
--
作者：
Levent Sagun;L. Bottou;Yann LeCun
我们观察训练前后损失函数的Hessian特征值。特征值分布由两部分组成，集中在零附近的大块和分散在远离零的边缘。我们提出了大量的经验证据，表明系统是如何过度参数化的，以及依赖于输入数据的边缘。
We look at the eigenvalues of the Hessian of a loss function before and after training. The eigenvalue distribution is seen to be composed of two parts, the bulk which is concentrated around zero, and the edges which are scattered away from zero. We present empirical evidence for the bulk indicating how over-parametrized the system is, and for the edges that depend on the input data.