The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry
The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry
复制标题
DOI：
--
复制
发表时间：
2020-06
期刊：
影响因子：
--
通讯作者：
Tomohiro Hayase;Ryo Karakida
中科院分区：
文献类型：
--
作者：
Tomohiro Hayase;Ryo Karakida
Fisher信息矩阵(FIM)描述了参数空间的局部度量，是理解深度神经网络(DNN)可训练性的基础。通过聚焦于实现动态等距的全连通网络，我们研究了给定单一输入的FIM的谱分布。然后，虽然已知的动态等距使特定的反向传播信号与深度无关，但我们发现参数空间的局部度量依赖于深度。特别地，我们得到了给定单输入的FIM谱的精确表达式，并揭示了它集中在深度点附近。在这里，考虑到随机初始化和宽极限，我们基于自由概率理论构造了一种检查频谱的代数方法，它是随机矩阵理论的代数包装。作为副产品，我们给出了双隐层情况下的可解谱分布。最后，我们实证地证实了小批量FIM的谱与单输入的谱具有相同的性质。实验结果表明，在DNN在线训练的初始阶段，FIM对深度的依赖决定了适合收敛的学习率的大小。
The Fisher information matrix (FIM) is fundamental for understanding the trainability of deep neural networks (DNN) since it describes the local metric of the parameter space. We investigate the spectral distribution of the FIM given a single input by focusing on fully-connected networks achieving dynamical isometry. Then, while dynamical isometry is known to keep specific backpropagated signals independent of the depth, we find that the parameter space's local metric depends on the depth. In particular, we obtain an exact expression of the spectrum of the FIM given a single input and reveal that it concentrates around the depth point. Here, considering random initialization and the wide limit, we construct an algebraic methodology to examine the spectrum based on free probability theory, which is the algebraic wrapper of random matrix theory. As a byproduct, we provide the solvable spectral distribution in the two-hidden-layer case. Lastly, we empirically confirm that the spectrum of FIM with small batch-size has the same property as the single-input version. An experimental result shows that FIM's dependence on the depth determines the appropriate size of the learning rate for convergence at the initial phase of the online training of DNNs.