Asymptotics of Ridge Regression in Convolutional Models
Asymptotics of Ridge Regression in Convolutional Models
复制标题
DOI：
--
复制
发表时间：
2021-03
期刊：
影响因子：
--
通讯作者：
Mojtaba Sahraee-Ardakan;Tung Mai;Anup B. Rao;Ryan A. Rossi;S. Rangan;A. Fletcher
中科院分区：
文献类型：
--
作者：
Mojtaba Sahraee-Ardakan;Tung Mai;Anup B. Rao;Ryan A. Rossi;S. Rangan;A. Fletcher
近年来，对线性和广义线性模型等简单模型估计量的推广和估计误差的理解引起了人们的广泛关注。这在一定程度上是由于机器学习社区中的一个有趣的观察，即高度过度参数化的神经网络实现了零训练错误，但它们能够在测试样本上很好地泛化。这种现象被所谓的双下降曲线捕获，其中泛化误差在插值阈值之后再次开始下降。最近的一系列工作试图用简单的模型来解释这种现象。在这项工作中，我们分析了卷积线性模型的岭估计的估计误差的渐近性。这些卷积逆问题，也称为反卷积，自然出现在不同的领域，如地震学，成像和声学等。我们的结果适用于一个大类的输入分布，包括独立同分布。作为一个特殊的案例。我们推导出精确的估计误差的岭估计，在一定的高维政权。我们在卷积模型的实验中展示了双下降现象，并表明我们的理论结果与实验相匹配。
Understanding generalization and estimation error of estimators for simple models such as linear and generalized linear models has attracted a lot of attention recently. This is in part due to an interesting observation made in machine learning community that highly over-parameterized neural networks achieve zero training error, and yet they are able to generalize well over the test samples. This phenomenon is captured by the so called double descent curve, where the generalization error starts decreasing again after the interpolation threshold. A series of recent works tried to explain such phenomenon for simple models. In this work, we analyze the asymptotics of estimation error in ridge estimators for convolutional linear models. These convolutional inverse problems, also known as deconvolution, naturally arise in different fields such as seismology, imaging, and acoustics among others. Our results hold for a large class of input distributions that include i.i.d. features as a special case. We derive exact formulae for estimation error of ridge estimators that hold in a certain high-dimensional regime. We show the double descent phenomenon in our experiments for convolutional models and show that our theoretical results match the experiments.