The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network
The sample complexity of pattern classification with neural networks: The size of the weights is more important than the size of the network
复制标题
DOI：
10.1109/18.661502
复制
发表时间：
1998-03-01
影响因子：
2.5
通讯作者：
Bartlett, PL
中科院分区：
文献类型：
--
作者：
Bartlett, PL
计算学习理论的样本复杂性结果，当应用于模式分类问题的神经网络学习时，表明为了获得良好的泛化性能，训练示例的数量应至少与网络中可调整参数的数量线性增长。本文的结果表明，如果将大型神经网络用于模式分类问题，并且学习算法找到一个具有较小权重的网络，该网络在训练模式上具有较小的平方误差，则泛化性能取决于权重的大小而不是权重的数量，例如，考虑两层sigmoid 单元的前馈网络，其中与每个单元相关的权重大小之和以 A 为界，输入维度为 n，我们表明误分类概率不超过某个误差估计（与训练集上的平方误差有关）加上 A(3) root(log n)/m（忽略 log A 和 log m 因子），其中 m 是训练模式的数量，这可以解释神经网络的泛化性能，特别是当训练数量较多时示例的数量远小于权重的数量，它还支持尝试在训练期间保持权重较小的启发式方法（例如权重衰减和早期停止），证明技术似乎对于其他模式分类器的分析很有用：当输入域是完全有界的度量空间时，我们使用相同的方法为决策边界远离训练示例的分类器给出错误分类概率的上限。
Sample complexity results from computational learning theory, when applied to neural network learning for pattern classification problems, suggest that for good generalization performance the number of training examples should grow at least linearly with the number of adjustable parameters in the network, Results in this paper show that if a large neural network is used for a pattern classification problem and the learning algorithm finds a network with small weights that has small squared error on the training patterns, then the generalization performance depends on the size of the weights rather than the number of weights, For example, consider a two-layer feedforward network of sigmoid units, in which the sum of the magnitudes of the weights associated with each unit is bounded by A and the input dimension is n, We show that the misclassification probability is no more than a certain error estimate (that is related to squared error on the training set) plus A(3) root(log n)/m (ignoring log A and log m factors), where m is the number of training patterns, This may explain the generalization performance of neural networks, particularly when the number of training examples is considerably smaller than the number of weights, It also supports heuristics (such as weight decay and early stopping) that attempt to keep the weights small during training, The proof techniques appear to be useful for the analysis of other pattern classifiers: when the input domain is a totally bounded metric space, we use the same approach to give upper bounds on misclassification probability for classifiers with decision boundaries that are far from the training examples.