The Recurrent Neural Tangent Kernel
The Recurrent Neural Tangent Kernel
复制标题
DOI：
--
复制
发表时间：
2020-06
期刊：
影响因子：
--
通讯作者：
Sina Alemohammad;Zichao Wang;Randall Balestriero;Richard Baraniuk
中科院分区：
文献类型：
--
作者：
Sina Alemohammad;Zichao Wang;Randall Balestriero;Richard Baraniuk
通过所谓的神经切线内核（NTK）方法，对无限宽度极限的深网（DNS）的研究为学习，概括和初始化的影响提供了新的见解。一个关键的DN架构仍有待核，即经常性神经网络（RNN）。在本文中，我们介绍并研究了复发性神经切线内核（RNTK），该内部有新的见解对过份术的RNN的行为，包括RNTK如何加权不同的时间步长以在不同的初始化参数和非线性选择和非线性选择以及非线性选择和非线性选择和非线性选择下形成输出。如何处理不同长度的输入。我们通过许多实验证明，RNTK比其他内核提供了显着的性能增长，包括一系列不同数据集的标准NTK。 RNTK的一个独特好处是，它与输入的长度不可知，与其他内核形成鲜明对比。
The study of deep networks (DNs) in the infinite-width limit, via the so-called Neural Tangent Kernel (NTK) approach, has provided new insights into the dynamics of learning, generalization, and the impact of initialization. One key DN architecture remains to be kernelized, namely, the Recurrent Neural Network (RNN). In this paper we introduce and study the Recurrent Neural Tangent Kernel (RNTK), which sheds new insights into the behavior of overparametrized RNNs, including how different time steps are weighted by the RNTK to form the output under different initialization parameters and nonlinearity choices, and how inputs of different lengths are treated. We demonstrate via a number of experiments that the RNTK offers significant performance gains over other kernels, including standard NTKs across a range of different data sets. A unique benefit of the RNTK is that it is agnostic to the length of the input, in stark contrast to other kernels.