Replica analysis of overfitting in generalized linear regression models
Replica analysis of overfitting in generalized linear regression models
复制标题
DOI：
10.1088/1751-8121/aba028
复制
发表时间：
2020-09-11
影响因子：
2.1
通讯作者：
Antenucci, F.
中科院分区：
文献类型：
--
作者：
Coolen, A. C. C.;Sheikh, M.;Antenucci, F.
几乎所有的统计推断方法都是针对数据样本数N远大于数据维度p的情况开发的。如果p = O（N），则由于过拟合，最大似然（ML）或最大后验概率（MAP）等推断协议是不可靠的。这种局限性已经对于许多学科日益高维的数据成为一个严重的瓶颈。我们最近表明，在时间-事件数据的考克斯回归中，过拟合误差不仅是噪声，而且主要是偏差的形式，以及如何使用统计物理学的复制方法来建模和预测这种偏差和噪声统计。在这里，我们将我们的方法扩展到任意广义线性回归模型（GLM），可能相关的协变量。我们分析过拟合ML/MAP推理，而不必指定数据类型或回归模型，只依赖于GLM形式，并推导出通用的顺序参数方程的情况下，L2先验。其次，我们推导出GLM中真实和推断回归系数之间的概率关系，并表明，对于相关的超参数尺度和相关协变量，L2正则化会导致系数向量的可预测方向变化。我们的研究结果，说明应用线性，逻辑，和考克斯回归，使一个正确的ML和MAP的GLM推断系统过拟合偏差，从而扩展到迄今为止禁止制度p=O（N）的适用性。
Nearly all statistical inference methods were developed for the regime where the number N of data samples is much larger than the data dimension p. Inference protocols such as maximum likelihood (ML) or maximum a posteriori probability (MAP) are unreliable if p = O(N), due to overfitting. This limitation has for many disciplines with increasingly high-dimensional data become a serious bottleneck. We recently showed that in Cox regression for time-to-event data the overfitting errors are not just noise but take mostly the form of a bias, and how with the replica method from statistical physics one can model and predict this bias and the noise statistics. Here we extend our approach to arbitrary generalized linear regression models (GLM), with possibly correlated covariates. We analyse overfitting in ML/MAP inference without having to specify data types or regression models, relying only on the GLM form, and derive generic order parameter equations for the case of L2 priors. Second, we derive the probabilistic relationship between true and inferred regression coefficients in GLMs, and show that, for the relevant hyperparameter scaling and correlated covariates, the L2 regularization causes a predictable direction change of the coefficient vector. Our results, illustrated by application to linear, logistic, and Cox regression, enable one to correct ML and MAP inferences in GLMs systematically for overfitting bias, and thus extend their applicability into the hitherto forbidden regime p=O(N).