Learning efficiency of redundant neural networks in Bayesian estimation
Learning efficiency of redundant neural networks in Bayesian estimation
复制标题
贝叶斯估计中冗余神经网络的学习效率
DOI：
10.1109/72.963783
复制
发表时间：
2001
影响因子：
--
通讯作者：
Sumio Watanabe
中科院分区：
文献类型：
--
作者：
Sumio Watanabe
证明了层状神经网络的贝叶斯随机复杂度在包含真实分布的情况下渐近小于正则统计模型的贝叶斯随机复杂度。我们考虑一种情况，当一个具有M个输入单元，H个隐藏单元和N个输出单元的三层感知器被训练来估计具有H(0)个隐藏单元的模型所表示的真实分布，并证明随机复杂度渐近小于（1/2）{H(0) (M+N)+R} log N，其中N是训练样本的数量，R是H-H(0)， M和N的函数，远远小于冗余参数的数量。由于贝叶斯估计的泛化误差等于随机复杂度的增加，所以如果有渐近展开式，它小于（1/2n） {H(0) (M+N)+R}。在此基础上，从统计学的角度讨论了分层神经网络与规则统计模型的区别。
This paper proves that the Bayesian stochastic complexity of a layered neural network is asymptotically smaller than that of a regular statistical model if it contains the true distribution. We consider a case when a three-layer perceptron with M input units, H hidden units and N output units is trained to estimate the true distribution represented by the model with H(0) hidden units and prove that the stochastic complexity is asymptotically smaller than (1/2) {H(0) (M+N)+R} log n where n is the number of training samples and R is a function of H-H(0), M, and N that is far smaller than the number of redundant parameters. Since the generalization error of Bayesian estimation is equal to the increase of stochastic complexity, it is smaller than (1/2n) {H(0) (M+N)+R} if it has an asymptotic expansion. Based on the results, the difference between layered neural networks and regular statistical models is discussed from the statistical point of view.