Fast Adaptation with Linearized Neural Networks
Fast Adaptation with Linearized Neural Networks
复制标题
DOI：
--
复制
发表时间：
2021-03
期刊：
影响因子：
--
通讯作者：
Wesley J. Maddox;Shuai Tang;Pablo G. Moreno;A. Wilson;Andreas C. Damianou
中科院分区：
文献类型：
--
作者：
Wesley J. Maddox;Shuai Tang;Pablo G. Moreno;A. Wilson;Andreas C. Damianou
训练好的神经网络的感应偏差很难理解，因此很难适应新的环境。我们研究了神经网络线性化的感应偏差，这是对完整网络函数的令人惊讶的好总结。受这一发现的启发，我们提出了一种通过由网络的雅可比设计的核来将这些归纳偏差嵌入到高斯过程中的技术。在这种情况下，领域适应采用可解释的后验推断的形式，并伴随着不确定性估计。这种推理是分析性的，不存在标准技术中的局部最优问题，例如将神经网络权重微调到新任务。我们开发了基于矩阵乘法的显著计算加速，包括可伸缩Fisher向量积的一种新实现。我们在图像分类和回归上的实验表明，与神经网络微调相比，该框架在转移学习方面具有良好的前景和便利性。代码可在https://github.com/amzn/xfer/tree/master/finite_ntk.上找到
The inductive biases of trained neural networks are difficult to understand and, consequently, to adapt to new settings. We study the inductive biases of linearizations of neural networks, which we show to be surprisingly good summaries of the full network functions. Inspired by this finding, we propose a technique for embedding these inductive biases into Gaussian processes through a kernel designed from the Jacobian of the network. In this setting, domain adaptation takes the form of interpretable posterior inference, with accompanying uncertainty estimation. This inference is analytic and free of local optima issues found in standard techniques such as fine-tuning neural network weights to a new task. We develop significant computational speed-ups based on matrix multiplies, including a novel implementation for scalable Fisher vector products. Our experiments on both image classification and regression demonstrate the promise and convenience of this framework for transfer learning, compared to neural network fine-tuning. Code is available at https://github.com/amzn/xfer/tree/master/finite_ntk.