Benign overfitting in linear regression
Benign overfitting in linear regression
复制标题
DOI：
10.1073/pnas.1907378117
复制
发表时间：
2020-12-01
影响因子：
11.1
通讯作者：
Tsigler, Alexander
中科院分区：
文献类型：
--
作者：
Bartlett, Peter L.;Long, Philip M.;Tsigler, Alexander
良性过度拟合的现象是深度学习方法论所发现的关键奥秘之一：即使非常适合嘈杂的训练数据，深度神经网络似乎也可以很好地预测。在这种现象中，我们考虑何时与线性回归中的训练数据完全合适，与准确的预测兼容。我们给出了线性回归问题的表征，最小规范插值预测规则的预测准确性几乎是最佳的。该表征是根据数据协方差的有效等级的两个概念。它表明，在这种情况下，过度参数化对于良性过度拟合至关重要：参数空间中不重要的预测的方向数必须显着超过样本量。通过研究该表征表明的数据协方差属性的示例，我们发现有限维数据的重要作用：最小规范插值插值预测规则的准确性是最佳的准确性，即最佳的准确性，范围更窄当数据位于无限维空间中的数据分布与数据位于有限维空间中的尺寸相比，比样本量更快的数据分布。
The phenomenon of benign overfitting is one of the key mysteries uncovered by deep learning methodology: deep neural networks seem to predict well, even with a perfect fit to noisy training data. Motivated by this phenomenon, we consider when a perfect fit to training data in linear regression is compatible with accurate prediction. We give a characterization of linear regression problems for which the minimum norm interpolating prediction rule has near-optimal prediction accuracy. The characterization is in terms of two notions of the effective rank of the data covariance. It shows that overparameterization is essential for benign overfitting in this setting: the number of directions in parameter space that are unimportant for prediction must significantly exceed the sample size. By studying examples of data covariance properties that this characterization shows are required for benign overfitting, we find an important role for finite-dimensional data: the accuracy of the minimum norm interpolating prediction rule approaches the best possible accuracy for a much narrower range of properties of the data distribution when the data lie in an infinite-dimensional space vs. when the data lie in a finite-dimensional space with dimension that grows faster than the sample size.