On Optimal Generalizability in Parametric Learning
On Optimal Generalizability in Parametric Learning
复制标题
参数学习中的最优泛化性
DOI：
--
复制
发表时间：
2017
期刊：
影响因子：
--
通讯作者：
V. Tarokh
中科院分区：
文献类型：
--
作者：
Ahmad Beirami;Meisam Razaviyayn;Shahin Shahrampour;V. Tarokh
我们考虑参数学习问题，其中学习者的目标由一个参数损失函数决定。利用经验风险最小化和可能的正则化，推断的参数向量将偏向于训练样本。在实践中，通过交叉验证过程来测量这种偏差，其中数据集被划分为用于训练的训练集和验证集，该验证集不在训练中使用，而是留下来测量样本外性能。经典的交叉验证策略是留一交叉验证(LOOCV)，其中一个样本被遗漏用于验证，而对呈现给学习者的其余样本进行训练，并在所有样本上重复这一过程。LOOCV由于计算复杂度较高，在实际应用中很少使用。本文首先提出了一种计算高效的近似LOOCV(ALOOCV)，并为其性能提供了理论保证。然后利用ALOOCV给出了在经验风险最小化框架下寻找正则化函数的优化算法。在数值实验中，我们展示了ALOOCV的精度和效率，以及我们提出的正则化优化框架。
We consider the parametric learning problem, where the objective of the learner is determined by a parametric loss function. Employing empirical risk minimization with possibly regularization, the inferred parameter vector will be biased toward the training samples. Such bias is measured by the cross validation procedure in practice where the data set is partitioned into a training set used for training and a validation set, which is not used in training and is left to measure the out-of-sample performance. A classical cross validation strategy is the leave-one-out cross validation (LOOCV) where one sample is left out for validation and training is done on the rest of the samples that are presented to the learner, and this process is repeated on all of the samples. LOOCV is rarely used in practice due to the high computational complexity. In this paper, we first develop a computationally efficient approximate LOOCV (ALOOCV) and provide theoretical guarantees for its performance. Then we use ALOOCV to provide an optimization algorithm for finding the regularizer in the empirical risk minimization framework. In our numerical experiments, we illustrate the accuracy and efficiency of ALOOCV as well as our proposed framework for the optimization of the regularizer.