Generalization Error of Generalized Linear Models in High Dimensions
Generalization Error of Generalized Linear Models in High Dimensions
复制标题
DOI：
--
复制
发表时间：
2020-05
期刊：
影响因子：
--
通讯作者：
M Motavali Emami;Mojtaba Sahraee-Ardakan;Parthe Pandit;S. Rangan;A. Fletcher
中科院分区：
文献类型：
--
作者：
M Motavali Emami;Mojtaba Sahraee-Ardakan;Parthe Pandit;S. Rangan;A. Fletcher
机器学习的核心问题是学习规则在以前看不见的数据上的可推广性。虽然基于神经网络的过参数化模型现在在机器学习应用中无处不在，但我们对其泛化能力的理解是不完整的。由于底层学习问题的非凸性，这一任务变得更加困难。我们提供了一个一般框架来表征单层神经网络的渐近泛化误差（即，广义线性模型）与任意非线性，使其适用于回归以及分类问题。该框架能够分析（i）建模过程中的过度参数化和非线性;以及（ii）学习过程中损失函数，初始化和正则化的选择。我们的模型还捕获了训练和测试分布之间的不匹配。作为例子，我们分析了几个特殊的情况，即线性回归和逻辑回归。我们也能够严格地和分析地解释广义线性模型中的\n {双下降}现象。
At the heart of machine learning lies the question of generalizability of learned rules over previously unseen data. While over-parameterized models based on neural networks are now ubiquitous in machine learning applications, our understanding of their generalization capabilities is incomplete. This task is made harder by the non-convexity of the underlying learning problems. We provide a general framework to characterize the asymptotic generalization error for single-layer neural networks (i.e., generalized linear models) with arbitrary non-linearities, making it applicable to regression as well as classification problems. This framework enables analyzing the effect of (i) over-parameterization and non-linearity during modeling; and (ii) choices of loss function, initialization, and regularizer during learning. Our model also captures mismatch between training and test distributions. As examples, we analyze a few special cases, namely linear regression and logistic regression. We are also able to rigorously and analytically explain the \emph{double descent} phenomenon in generalized linear models.