Neural network learning with regulatizers and generalization ability

从理论和实验两方面阐明了正则化神经网络学习与泛化能力之间的关系。在本文的理论研究中，考虑了拉普拉斯正则化、高斯正则化以及它们的组合。在第一阶段，从理论上阐明了作者提出的具有遗忘的结构学习的各种经验步骤。例如，在假设场景除了少量不连续点外几乎是光滑的情况下，选择性遗忘的结构学习与视觉中的线过程有关。这是多元回归模型中最简单的情况。第三阶段对多元回归模型中泛化误差公式的过度简化进行了修正。因此，假设输入变量是相互独立的，并且真实的模型参数和噪声方差是先验已知的。我们修改了公式，以便允许输入变量之间的相关性。我们提出了一种基于数据从理论上评估正则化的新方法。首先，我们从数据中估计模型参数和噪声方差。其次，在假设这些估计为真的情况下，利用前面提出的方法计算最优正则化参数和模型参数。希望它能提供他们更好的估计。第三，假设所得估计为真，我们再次计算最优正则化参数和模型参数。这个过程可以反复重复。这种迭代估计是本研究的一个关键思想。对实际数据的应用表明，该方法以较小的泛化误差成功地获得了较好的估计。

The relation between neural network learning with regularizes and generalization ability is clarified both theoretically and empirically. In the present theoretical study, a Laplacian regularize, a Gaussian regularize and their combinations are considered.In the first stage, various empirical procedures in a structural learning with forgetting proposed by the authors are theoretically clarified. For example, a structural learning with selective forgetting has something to do with the line process in vision under the assumption that a scene is almost smooth except a small number of discontinuous points.In the second stage, a estimation of mean value using regularization is theoretically studied. It is the simplest case of multiple regression models. It is demonstrated that the proposed regularization method is effectively applied to real data.In the third stage, excessive simplification in a formulation of generalization errors in multiple regression models is rectified. So for it has been assumed that input variables are mutually independent, and true model parameters and a noise variance are known a priori. We modified formulations so as to allow correlations between input variables. We propose a novel procedure for theoretically evaluating regularizes based on data. Firstly, we estimate model parameters and a noise variance from data. Secondly, assuming that these estimates are true, we calculate the optimal regularization parameters and model parameters by the previously proposed method. It provides, hopefully, their better estimates. Thirdly, assuming that the resulting estimates are true, we again calculate the optimal regularization parameters and model parameters. This procedure can be repeated iteratively. This iterative estimation is a key idea of the present study. Applications of the proposed method to real data demonstrates that better estimates with smaller generalization errors are obtained successfully.