Generalization Capability of Memorization Leaning

监督学习的目的是能够正确地回答不一定包括在训练样本中的查询，即获得更高水平的泛化能力。然而，大多数的学习方法，如误差反向传播，都是所谓的记忆学习，其目的只是为了减少训练样本的误差。因此，没有最优概括的理论保证。这就产生了以下问题：一是澄清为什么记忆学习可以获得更高水平的概括能力，尽管它不需要概括能力。第二个问题是明确记忆学习的适用范围：记忆学习有效。三是进一步扩大适用范围的方法。对于第一个问题，我们引入了可采性的概念，给出了清晰的解释。对于第二个…更重要的是，我们引入了一族投影学习的概念，这使得我们可以从理论上同时讨论无限多种学习方法。利用可容许性和一族投影学习的概念，阐明了狭义记忆学习对于一族投影学习的适用范围。对于第三个问题，我们证明了有大量的解决方案：我们将记忆学习的概念从死记硬背扩展到纠错记忆学习，进一步扩大了适用范围。从主动学习的角度出发，给出了最大限度提高泛化能力的训练实例设计方法。此外，从模型选择的角度出发，提出了子空间信息准则(SIC)，这是一种模型选择准则，在理论上保证了它对有限数量训练样本的有效性。基于SIC，我们给出了最优正则化参数的一种设计方法。较少

The purpose of supervised learning is to be able to answer correctly to queries that are not necessarily included in the training examples, i.e., to acquire a higher level of the generalization capability. However, most of the learning methods such as the error back-propagation are the so-called memorization learning, which is aimed at reducing the error only for training examples. Therefore, there is no theoretical guarantee for optimal generalization.This gives rise to the following problems: First is to clarify the reason why a higher level of the generalization capability can be acquired by the memorization learning despite the fact that it does not require the generalization capability. The second problem is to clarify the range of applicability that: the memorization learning works effectively. Third is to develop methods for further expanding the range of applicability.For the first problem, we gave a lucid explanation by introducing the concept of admissibility. For the second … More problem, we introduced the concept of a family of projection learnings which allows us to theoretically discuss an infinitely many kinds of learning methods simultaneously. Utilizing the concepts of admissibility and a family of projection learning, we clarified the range of applicability of the memorization learning in the narrow sense with respect to a family of projection learnings. For the third problem, we showed that there are a large number of solutions : We extended the concept of the memorization learning from the rote memorization learning to the error corrected memorization learning, which further enlarges the range of applicability. From the view point of active learning, we gave design methods of training examples that maximally enhance the generalization capability. Furthermore, from the standpoint of model selection, we proposed the subspace information criterion (SIC) , which is a model selection criterion with its effectiveness theoretically guaranteed for a finite number of training examples. Based on SIC, we gave, for example, a design method of the optimal regularization parameter. Less