Characterizing the Spectrum of the NTK via a Power Series Expansion
Characterizing the Spectrum of the NTK via a Power Series Expansion
复制标题
DOI：
10.48550/arxiv.2211.07844
复制
发表时间：
2022-11
期刊：
影响因子：
--
通讯作者：
Michael Murray;Hui Jin;Benjamin Bowman;Guido Montúfar
中科院分区：
文献类型：
--
作者：
Michael Murray;Hui Jin;Benjamin Bowman;Guido Montúfar
在网络初始化的温和条件下，我们推导出无限宽度限制下任意深度前馈网络的神经正切核（NTK）的幂级数展开。我们提供了该幂级数的系数表达式，该表达式取决于激活函数的 Hermite 系数以及网络的深度。我们观察到 Hermite 系数的快速衰减导致 NTK 系数的快速衰减，并探索了深度的作用。使用这个系列，首先我们将 NTK 的有效秩与输入数据 Gram 的有效秩相关联。其次，对于均匀绘制在球体上的数据，我们研究 NTK 的特征值，分析激活函数选择的影响。最后，对于具有足够快 Hermite 系数衰减的通用数据和激活函数，我们推导出 NTK 谱的渐近上限。
Under mild conditions on the network initialization we derive a power series expansion for the Neural Tangent Kernel (NTK) of arbitrarily deep feedforward networks in the infinite width limit. We provide expressions for the coefficients of this power series which depend on both the Hermite coefficients of the activation function as well as the depth of the network. We observe faster decay of the Hermite coefficients leads to faster decay in the NTK coefficients and explore the role of depth. Using this series, first we relate the effective rank of the NTK to the effective rank of the input-data Gram. Second, for data drawn uniformly on the sphere we study the eigenvalues of the NTK, analyzing the impact of the choice of activation function. Finally, for generic data and activation functions with sufficiently fast Hermite coefficient decay, we derive an asymptotic upper bound on the spectrum of the NTK.