An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
复制标题
通过 Hessian 特征值密度进行神经网络优化的研究
DOI：
--
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
Ying Xiao
中科院分区：
文献类型：
--
作者：
B. Ghorbani;Shankar Krishnan;Ying Xiao
为了了解深度神经网络优化的动态，我们开发了一种工具来研究整个优化过程中整个 Hessian 谱的演化。利用这一点，我们研究了深度学习文献中有关平滑度、曲率和锐度的一些假设。然后，我们彻底分析了光谱的一个关键结构特征：在非批量归一化网络中，我们观察到光谱中大的孤立特征值的快速出现，以及相应特征空间中梯度的惊人集中。在批量归一化网络中，这两种效应几乎不存在。我们描述了这些影响，并通过理论和实验解释它们如何影响优化速度。作为这项工作的一部分，我们采用了数值线性代数的先进工具，可以对 ImageNet 规模神经网络的整个 Hessian 谱进行可扩展且准确的估计；该技术可能在其他应用中具有独立的意义。
To understand the dynamics of optimization in deep neural networks, we develop a tool to study the evolution of the entire Hessian spectrum throughout the optimization process. Using this, we study a number of hypotheses concerning smoothness, curvature, and sharpness in the deep learning literature. We then thoroughly analyze a crucial structural feature of the spectra: in non-batch normalized networks, we observe the rapid appearance of large isolated eigenvalues in the spectrum, along with a surprising concentration of the gradient in the corresponding eigenspaces. In batch normalized networks, these two effects are almost absent. We characterize these effects, and explain how they affect optimization speed through both theory and experiments. As part of this work, we adapt advanced tools from numerical linear algebra that allow scalable and accurate estimation of the entire Hessian spectrum of ImageNet-scale neural networks; this technique may be of independent interest in other applications.