Study about a construction of optimally generalizing neural networks

在这项研究中，我们首先将神经网络的训练问题形式化为函数逼近反问题之一。其次，根据隐含单元的个数、基函数和与每个隐含单元相连的权重，给出了泛化能力最优的充要条件。此外，我们还给出了一种构造具有最优泛化能力的神经网络的方法，从这个方法可以看出，具有相同泛化能力的神经网络是无限多的。从这个无穷大的数字中，我们指定了对于实际使用中可能出现的某些类型的故障最健壮的。具体地说，我们给出了在神经网络中确定权值的方法，这些方法最优地抑制了以下每个故障的影响：权值中的比例误差、连接故障和固定伽马故障。其次，我们构造了一种增量学习方法，该方法在保持神经网络对已学习的所有数据都是最优泛化的基础上，在每一步只使用当前网络和一个新的数据来获得一个新网络。我们认为我们的结果具有应用于主动学习问题的潜力，此外，我们还给出了使用误差反向传播算法训练神经网络时出现的过度学习问题的解决方案，即我们引入了由两个学习准则之间的关系定义的容许性的概念。根据可接受性，我们设计了选择训练数据的方法，以防止过度学习。

In this sutdy, we first formalized the problem of training a neural network as one of an inverse problem in function approximation. Next, we provided necessary and sufficient conditions for optimal generalization capability in terms of the number of hidden units, the basis functions, and the weights connected to each hidden unit. Furthermore, we gave a methodology to construct neural networks with optimal generalization capability.From this methodology, we can see that are an infinite numbers of neural networks with the same generalization capability. From among this infinite number, we specified the ones which are most robust with respect to some kinds of faults which may occur in actual usage. Concretely, we gave methods to decide weights in neural networks which optimally suppress the influences of each of the following faults : a proportional errors in the weights, a connection fault, and a stuck-at gamma fault. Moreover, we gave the methods to decide not only weights but also basis functions for the hidden units which optimally suppress the above three faults.Next we constructed a method for incremental learning in which only the current network and one new datum are used to obtain a new network at each step, while maintaining the property that the neural network is optimally generalizing with respect to all of the data learned so far. We think our results have the potential for being applied to the problem of active learning.Moreover, we gave a solution to the problem of over-learning which occurs in training of neural networks using the error-backpropagation algorithm, i.e., we introduced the concept of admissibility defined by relation-ship between two learning criteria. According to the admissibility, we devised methods for choosing training data to prevent over-leaning.