Weighted neural tangent kernel: a generalized and improved network-induced kernel
Weighted neural tangent kernel: a generalized and improved network-induced kernel
复制标题
加权神经正切核：一种广义和改进的网络诱导核
DOI：
10.1007/s10994-023-06356-3
复制
发表时间：
2021
期刊：
影响因子：
7.5
通讯作者：
Xiaolin Huang
中科院分区：
文献类型：
--
作者：
Lei Tan;Shutong Wu;Wenxing Zhou;Xiaolin Huang
神经元切线内核（NTK）最近吸引了激烈的研究，因为它描述了通过梯度下降训练的过度参数化神经元网络（NN）的演变。对于NNS，可以部分解释NTK回归估算器的实际性能。改进的工具可以在可调梯度下降方向下捕获过度参数的NN训练动力学。 WNTK回归估计器与相应的NN估计器之间具有不同参数的学习速率。 NN下降方向可以通过多内核优化进行训练。
The neural tangent kernel (NTK) has recently attracted intense study, as it describes the evolution of an over-parameterized neural network (NN) trained by gradient descent. However, it is now well-known that gradient descent is not always a good optimizer for NNs, which can partially explain the unsatisfactory practical performance of the NTK regression estimator. In this paper, we introduce the weighted neural tangent kernel (WNTK), a generalized and improved tool, which can capture an over-parameterized NN’s training dynamics under adjusted gradient descent direction. Theoretically, in the infinite-width limit, we prove: (1) the stability of the WNTK at initialization and during training, and (2) the equivalence between the WNTK regression estimator and the corresponding NN estimator with different learning rates on different parameters. With the proposed weight update algorithm, weight terms, or equivalently NN descent directions, can be trained through multiple-kernel optimization. Both empirical and analytical WNTKs outperform the corresponding NTKs in numerical experiments, coinciding with the fact that adjusted gradient descent could outperform original gradient descent in NNs’ training.
登录
查看更多内容
DOI：
--
发表时间：
2018-02
期刊：
ArXiv
影响因子：
--
作者：
A. G. Matthews;Mark Rowland;Jiri Hron;Richard E. Turner;Zoubin Ghahramani
通讯作者：
A. G. Matthews;Mark Rowland;Jiri Hron;Richard E. Turner;Zoubin Ghahramani
DOI：
--
发表时间：
2020-06
期刊：
ArXiv
影响因子：
--
作者：
Sina Alemohammad;Zichao Wang;Randall Balestriero;Richard Baraniuk
通讯作者：
Sina Alemohammad;Zichao Wang;Randall Balestriero;Richard Baraniuk
DOI：
--
发表时间：
2019-09
期刊：
ArXiv
影响因子：
--
作者：
B. Hanin;M. Nica
通讯作者：
B. Hanin;M. Nica
影响因子：
3.9
作者：
Roman Novak;Lechao Xiao;Yasaman Bahri;Jaehoon Lee;Greg Yang;Jiri Hron;Daniel A. Abolafia;Jeffrey Pennin
通讯作者：
Roman Novak;Lechao Xiao;Yasaman Bahri;Jaehoon Lee;Greg Yang;Jiri Hron;Daniel A. Abolafia;Jeffrey Pennin
DOI：
--
发表时间：
2019-10
期刊：
ArXiv
影响因子：
--
作者：
Sanjeev Arora;S. Du;Zhiyuan Li;R. Salakhutdinov;Ruosong Wang;Dingli Yu
通讯作者：
Sanjeev Arora;S. Du;Zhiyuan Li;R. Salakhutdinov;Ruosong Wang;Dingli Yu