Precise learning curves and higher-order scaling limits for dot-product kernel regression
Precise learning curves and higher-order scaling limits for dot-product kernel regression
复制标题
DOI：
10.1088/1742-5468/ad01b7
复制
发表时间：
2022-05
期刊：
影响因子：
--
通讯作者：
Lechao Xiao;Jeffrey Pennington
中科院分区：
文献类型：
--
作者：
Lechao Xiao;Jeffrey Pennington
随着现代机器学习模型不断推进计算前沿，为不同模型和数据缩放机制下的预期性能改进制定准确的估计变得越来越重要。目前，对描述预测误差如何依赖于样本数量的学习曲线(LC)的理论理解局限于大样本渐近曲线(m→∞)，或者对于某些简单的数据分布，限于样本数量与维度(m∝d)成线性关系的高维渐近曲线。这两个机制之间存在着巨大的鸿沟，包括本文所要讨论的∝DR的所有高阶标度关系。研究了点积核函数的核岭回归问题，在m/dR不变的r阶渐近标度下，给出了均匀取自具有各向同性随机标号的球面数据的检验误差均值、偏差均值和方差均值的精确公式。当m≈dr/r时，我们在LC中观察到一个峰值！对于任何整数r，导致在多个尺度上的多次采样下降和非平凡行为。我们包括一个CoLab(可在https://tinyurl.com/2nzym7ym)笔记本上找到)，它复制了论文的基本结果。
As modern machine learning models continue to advance the computational frontier, it has become increasingly important to develop precise estimates for expected performance improvements under different model and data scaling regimes. Currently, theoretical understanding of the learning curves (LCs) that characterize how the prediction error depends on the number of samples is restricted to either large-sample asymptotics ( m→∞ ) or, for certain simple data distributions, to the high-dimensional asymptotics in which the number of samples scales linearly with the dimension ( m∝d ). There is a wide gulf between these two regimes, including all higher-order scaling relations m∝dr , which are the subject of the present paper. We focus on the problem of kernel ridge regression for dot-product kernels and present precise formulas for the mean of the test error, bias and variance, for data drawn uniformly from the sphere with isotropic random labels in the rth-order asymptotic scaling regime m→∞ with m/dr held constant. We observe a peak in the LC whenever m≈dr/r! for any integer r, leading to multiple sample-wise descent and non-trivial behavior at multiple scales. We include a colab (available at: https://tinyurl.com/2nzym7ym) notebook that reproduces the essential results of the paper.