Kernel Stability for Model Selection in Kernel-Based Algorithms
Kernel Stability for Model Selection in Kernel-Based Algorithms
复制标题
基于内核的算法中模型选择的内核稳定性
DOI：
10.1109/tcyb.2019.2923824
复制
发表时间：
2019
影响因子：
11.8
通讯作者：
Weiping Wang
中科院分区：
文献类型：
--
作者：
Yong Liu;Shizhong Liao;Hua Zhang;Wenqi Ren;Weiping Wang
模型选择是基于核的算法的基本问题之一，通常通过最小化泛化误差的估计来完成。学习机的稳定性和交叉验证（CV）误差的概念包括两个广泛使用的工具，用于分析泛化性能。然而，这两种工具在应用于模型选择时都存在一些缺点：1）由于难以估计其特定值，学习机的稳定性不实用; 2）基于CV的泛化误差估计通常具有相对较高的方差，因此容易过拟合。为了克服这两个限制，我们提出了一个新的概念的核稳定性（KS）推导的推广误差界和方差界的CV和KS的实际模型选择的应用提供了一个有效的方法。与现有的学习机稳定性概念不同，KS定义在核矩阵上，因此，它可以避免估计其值的困难。我们揭示了KS和流行的一致稳定性的学习算法之间的关系，并进一步提出了几个基于KS的泛化误差界和方差界的CV。通过最小化建议的界限，我们提出了两个新的基于KS的标准，可以确保良好的性能。最后，我们在许多基准数据上实证分析了所提出的标准的性能，这表明我们基于KS的标准是合理和有效的。
Model selection is one of the fundamental problems in kernel-based algorithms, which is commonly done by minimizing an estimation of generalization error. The notion of stability and cross-validation (CV) error of learning machines consists of two widely used tools for analyzing the generalization performance. However, there are some disadvantages to both tools when applied for model selection: 1) the stability of learning machines is not practical due to the difficulty of the estimation of its specific value and 2) the CV-based estimate of generalization error usually has a relatively high variance, so it is prone to overfitting. To overcome these two limitations, we present a novel notion of kernel stability (KS) for deriving the generalization error bounds and variance bounds of CV and provide an effective approach to the application of KS for practical model selection. Unlike the existing notions of stability of the learning machine, KS is defined on the kernel matrix; hence, it can avoid the difficulty of the estimation of its value. We manifest the relationship between the KS and the popular uniform stability of the learning algorithm, and further propose several KS-based generalization error bounds and variance bounds of CV. By minimizing the proposed bounds, we present two novel KS-based criteria that can ensure good performance. Finally, we empirically analyze the performance of the proposed criteria on many benchmark data, which demonstrates that our KS-based criteria are sound and effective.