Universal statistics of Fisher information in deep neural networks: mean field approach*
Universal statistics of Fisher information in deep neural networks: mean field approach*
复制标题
DOI：
10.1088/1742-5468/abc62e
复制
发表时间：
2020-12-01
影响因子：
2.4
通讯作者：
Amari, Shun-ichi
中科院分区：
文献类型：
--
作者：
Karakida, Ryo;Akaho, Shotaro;Amari, Shun-ichi
Fisher信息矩阵（FIM）是表示随机模型特征的基本量，包括深度神经网络（dnn）。本研究揭示了FIM的新统计数据，在广泛的dnn类别中是普遍的。为此，我们使用随机权重和大宽度限制，这使我们能够利用平均场理论。我们研究了FIM特征值的渐近统计量，发现大多数特征值接近于零，而最大特征值取一个很大的值。因为参数空间的景观是由FIM定义的，所以它在大多数维度上是局部平坦的，但在其他维度上是严重扭曲的。此外，我们还展示了所得统计在学习策略中的潜在用途。首先，诱导平坦度的小特征值可以连接到基于规范的泛化能力容量度量。其次，诱导畸变的最大特征值使我们能够定量地估计梯度方法收敛的适当大小的学习率。
The Fisher information matrix (FIM) is a fundamental quantity to represent the characteristics of a stochastic model, including deep neural networks (DNNs). The present study reveals novel statistics of FIM that are universal among a wide class of DNNs. To this end, we use random weights and large width limits, which enables us to utilize mean field theories. We investigate the asymptotic statistics of the FIM's eigenvalues and reveal that most of them are close to zero while the maximum eigenvalue takes a huge value. Because the landscape of the parameter space is defined by the FIM, it is locally flat in most dimensions, but strongly distorted in others. Moreover, we demonstrate the potential usage of the derived statistics in learning strategies. First, small eigenvalues that induce flatness can be connected to a norm-based capacity measure of generalization ability. Second, the maximum eigenvalue that induces the distortion enables us to quantitatively estimate an appropriately sized learning rate for gradient methods to converge.