Scaling description of generalization with number of parameters in deep learning
Scaling description of generalization with number of parameters in deep learning
复制标题
DOI：
10.1088/1742-5468/ab633c
复制
发表时间：
2020-02-01
影响因子：
2.4
通讯作者：
Wyart, Matthieu
中科院分区：
文献类型：
--
作者：
Geiger, Mario;Jacot, Arthur;Wyart, Matthieu
监督式深度学习涉及具有大量N个参数的神经网络的训练。对于足够大的N，在所谓的过参数化状态下，基本上可以拟合训练数据点。基于稀疏性的参数表明，当N增长超过某个阈值N*时，泛化误差会增加。相反，经验研究表明，在过度参数化状态下，泛化误差随着n的减小而减小。我们通过一个新的框架来解决这个悖论。我们依靠所谓的神经切线内核，它将大型神经网络与内核方法连接起来，以表明初始化导致有限大小的随机波动平行于f(N) - < f(N)>，平行于类似于神经网络输出函数f(N)的N-1/4，围绕其期望< f(N)>。这些影响分类的泛化误差epsilon(f(N))：在自然假设下，它以类似于N-1/2的幂律方式衰减到平台值epsilon（f(∞)）。这种描述在所谓的干扰过渡N = N*处失效。在这个阈值下，我们认为平行于f(N)平行于发散。这一结果对已知发生在N*处的测试误差尖峰给出了合理的解释。我们的结果得到了MNIST和CIFAR图像数据集的广泛实证观察的证实。我们的分析最后表明，在给定计算包络的情况下，使用几个中等规模的网络获得最小的泛化误差，刚好超过N*，并平均它们的输出。
Supervised deep learning involves the training of neural networks with a large number N of parameters. For large enough N, in the so-called over-parametrized regime, one can essentially fit the training data points. Sparsitybased arguments would suggest that the generalization error increases as N grows past a certain threshold N*. Instead, empirical studies have shown that in the over-parametrized regime, generalization error keeps decreasing with N. We resolve this paradox through a new framework. We rely on the so-called Neural Tangent Kernel, which connects large neural nets to kernel methods, to show that the initialization causes finite-size random fluctuations parallel to f(N) - < f(N)>parallel to similar to N-1/4 of the neural net output function f(N) around its expectation < f(N)>. These affect the generalization error epsilon(f(N)) for classification: under natural assumptions, it decays to a plateau value epsilon(f(infinity)) in a power-law fashion similar to N-1/2. This description breaks down at a so-called jamming transition N = N*. At this threshold, we argue that parallel to f(N)parallel to diverges. This result leads to a plausible explanation for the cusp in test error known to occur at N*. Our results are confirmed by extensive empirical observations on the MNIST and CIFAR image datasets. Our analysis finally suggests that, given a computational envelope, the smallest generalization error is obtained using several networks of intermediate sizes, just beyond N*, and averaging their outputs.