Semi-Parametric Inducing Point Networks and Neural Processes
Semi-Parametric Inducing Point Networks and Neural Processes
复制标题
DOI：
--
复制
发表时间：
2022-05
期刊：
影响因子：
--
通讯作者：
R. Rastogi;Yair Schiff;Alon Hacohen;Zhaozhi Li;I-Hsiang Lee;Yuntian Deng;M. Sabuncu;Volodymyr Kuleshov-V
中科院分区：
文献类型：
--
作者：
R. Rastogi;Yair Schiff;Alon Hacohen;Zhaozhi Li;I-Hsiang Lee;Yuntian Deng;M. Sabuncu;Volodymyr Kuleshov-V
我们引入了半参数诱导点网络(SPIN)，这是一种通用的结构，可以在推理时以高效的计算方式查询训练集。半参数结构通常比参数模型更紧凑，但它们的计算复杂性通常是二次的。相比之下，Spin通过受诱导点方法启发的数据点之间的交叉注意机制获得线性复杂性。查询大的训练集在元学习中可能特别有用，因为它解锁了额外的训练信号，但往往超过了现有模型的比例限制。我们使用SPIN作为诱导点神经过程的基础，这是一种概率模型，支持元学习中的大背景，并在现有模型失败的情况下获得高精度。在我们的实验中，SPIN减少了对内存的需求，提高了一系列元学习任务的准确性，并改进了在一个重要的实际问题--基因归属--上的最先进性能。
We introduce semi-parametric inducing point networks (SPIN), a general-purpose architecture that can query the training set at inference time in a compute-efficient manner. Semi-parametric architectures are typically more compact than parametric models, but their computational complexity is often quadratic. In contrast, SPIN attains linear complexity via a cross-attention mechanism between datapoints inspired by inducing point methods. Querying large training sets can be particularly useful in meta-learning, as it unlocks additional training signal, but often exceeds the scaling limits of existing models. We use SPIN as the basis of the Inducing Point Neural Process, a probabilistic model which supports large contexts in meta-learning and achieves high accuracy where existing models fail. In our experiments, SPIN reduces memory requirements, improves accuracy across a range of meta-learning tasks, and improves state-of-the-art performance on an important practical problem, genotype imputation.