Large scale analysis of generalization error in learning using margin based classification methods
Large scale analysis of generalization error in learning using margin based classification methods
复制标题
使用基于边缘的分类方法对学习中的泛化误差进行大规模分析
DOI：
10.1088/1742-5468/abbed5
复制
发表时间：
2020-10-01
影响因子：
2.4
通讯作者：
Yang, Qinglong
中科院分区：
文献类型：
--
作者：
Huang, Hanwen;Yang, Qinglong
大间隔分类器是流行的分类方法。我们推导出一个家庭的大利润率分类器的泛化误差的渐近表达式，在限制的样本大小n和尺寸p到无穷大与固定的比率α = n/p。这个家庭涵盖了广泛的常用的分类器，包括支持向量机，距离加权歧视，惩罚逻辑回归。我们的结果可以用来建立两类分离的相变边界。我们假设数据是从具有任意协方差结构的单个多元高斯分布生成的。我们探讨了协方差矩阵的两种特殊选择：尖峰种群模型和第一层权重随机的两层神经网络。我们用于推导封闭形式表达式的方法来自统计物理学，称为复制方法。我们的渐近结果匹配模拟已经当n，p是几百的顺序。对于两层神经网络，我们再现了最近观察到的“双下降”现象的几个分类模型。我们还讨论了一些统计见解，可以从这些分析得出。
Large-margin classifiers are popular methods for classification. We derive the asymptotic expression for the generalization error of a family of large-margin classifiers in the limit of both sample size n and dimension p going to infinity with fixed ratio alpha = n/p. This family covers a broad range of commonly used classifiers including support vector machine, distance weighted discrimination, and penalized logistic regression. Our result can be used to establish the phase transition boundary for the separability of two classes. We assume that the data are generated from a single multivariate Gaussian distribution with arbitrary covariance structure. We explore two special choices for the covariance matrix: spiked population model and two layer neural networks with random first layer weights. The method we used for deriving the closed-form expression is from statistical physics known as the replica method. Our asymptotic results match simulations already when n, p are of the order of a few hundreds. For two layer neural networks, we reproduce the recently observed 'double descent' phenomenology for several classification models. We also discuss some statistical insights that can be drawn from these analysis.