Bias of Estimators and Regularization Terms Noboru Murata
Bias of Estimators and Regularization Terms Noboru Murata
复制标题
估计量和正则化项的偏差 Noboru Murata
DOI：
--
复制
发表时间：
1998
期刊：
影响因子：
--
通讯作者：
Noboru Murata
中科院分区：
文献类型：
--
作者：
Noboru Murata
本文从最小化推广误差的角度讨论了正则化项（惩罚项）的作用。首先阐明了最小训练误差估计的偏差。偏差是由学习系统的非线性引起的，并且取决于训练样本的数量。然后，考虑到偏差和方差的平衡，考虑适当的正则化项的大小，使推广误差最小化。在这个框架中，正则化项的最佳大小是用损失函数的二阶和三阶导数来计算的。当学习系统有大量的可修改的参数，它是计算昂贵的计算高阶导数，因此，我们提出了一个简单的方法，通过广义AIC近似的最佳大小。
In this paper, a role of regularization terms (penalty terms) is discussed from the view point of minimizing the generalization error. First the bias of minimum training error estimation is clariied. The bias is caused by the nonlinearity of the learning system and depends on the number of training examples. Then an appropriate size of the regularization term is considered by taking account of the balance of the bias and the variance of the estimator, so that the generalization error is minimized. In this framework, the optimal size of the regularization term is calculated with the second and third order derivatives of the loss function. When the learning system has a large number of modiiable parameters, it is computationally expensive to calculate the higher order derivatives, thus we propose a simple method of approximating the optimal size via a generalized AIC.