Understanding Deep Learning (Still) Requires Rethinking Generalization
Understanding Deep Learning (Still) Requires Rethinking Generalization
复制标题
DOI：
10.1145/3446776
复制
发表时间：
2021-03-01
影响因子：
22.7
通讯作者：
Vinyals, Oriol
中科院分区：
文献类型：
--
作者：
Zhang, Chiyuan;Bengio, Samy;Vinyals, Oriol
尽管规模庞大，但成功的深度人工神经网络在训练和测试性能之间的差距非常小。传统的观点认为，小的泛化误差要么归因于模型族的属性，要么归因于训练过程中使用的正则化技术。通过广泛的系统实验，我们展示了这些传统方法无法解释为什么大型神经网络在实践中泛化良好。具体来说，我们的实验证明，用随机梯度方法训练的最先进的图像分类卷积网络很容易适应训练数据的随机标记。这种现象在定性上不受显式正则化的影响，即使我们用完全非结构化的随机噪声替换真实图像也会发生这种现象。我们用一个理论结构证实了这些实验结果，该理论结构表明，只要参数的数量超过数据点的数量，简单深度2神经网络就已经具有完美的有限样本表达能力，就像它在实践中通常做的那样。我们通过与传统模型的比较来解释我们的实验结果。我们在最后补充了一个新的部分，总结了自2000年以来该领域的最新进展。本文的原始版本。
Despite their massive size, successful deep artificial neural networks can exhibit a remarkably small gap between training and test performance. Conventional wisdom attributes small generalization error either to properties of the model family or to the regularization techniques used during training.Through extensive systematic experiments, we show how these traditional approaches fail to explain why large neural networks generalize well in practice. Specifically, our experiments establish that state-of-the-art convolutional networks for image classification trained with stochastic gradient methods easily fit a random labeling of the training data. This phenomenon is qualitatively unaffected by explicit regularization and occurs even if we replace the true images by completely unstructured random noise. We corroborate these experimental findings with a theoretical construction showing that simple depth two neural networks already have perfect finite sample expressivity as soon as the number of parameters exceeds the number of data points as it usually does in practice.We interpret our experimental findings by comparison with traditional models.We supplement this republication with a new section at the end summarizing recent progresses in the field since the original version of this paper.