The effects of adding noise during backpropagation training on a generalization performance
The effects of adding noise during backpropagation training on a generalization performance
复制标题
DOI：
10.1162/neco.1996.8.3.643
复制
发表时间：
1996-04-01
影响因子：
2.9
通讯作者：
An, GZ
中科院分区：
文献类型：
--
作者：
An, GZ
研究了在多层前馈神经网络的反向传播训练过程中，在输入、输出、权值连接和权值变化中加入噪声的影响。我们严格推导和分析了受噪声影响的训练过程中最小化的目标函数。我们表明，输入噪声和权值噪声分别促使神经网络输出成为输入或其权值的平滑函数。在弱噪声极限下，加入到神经网络输出中的噪声只会使目标函数改变一个常数。因此，它不能提高泛化。输入噪声在目标函数中引入惩罚项，这些惩罚项与正则化方法中发现的惩罚项相关，但又不同。对一个回归问题和一个分类问题进行了模拟，以进一步证实我们的分析。研究发现，输入噪声对提高这两个问题的泛化性能都是有效的。然而，权值噪声仅在分类问题上对提高泛化性能有效。其他形式的噪声实际上对泛化没有影响。
We study the effects of adding noise to the inputs, outputs, weight connections, and weight changes of multilayer feedforward neural networks during backpropagation training. We rigorously derive and analyze the objective functions that are minimized by the noise-affected training processes. We show that input noise and weight noise encourage the neural-network output to be a smooth function of the input or its weights, respectively. In the weak-noise limit, noise added to the output of the neural networks only changes the objective function by a constant. Hence, it cannot improve generalization. Input noise introduces penalty terms in the objective function that are related to, but distinct from, those found in the regularization approaches. Simulations have been performed on a regression and a classification problem to further substantiate our analysis. Input noise is found to be effective in improving the generalization performance for both problems. However, weight noise is found to be effective in improving the generalization performance only for the classification problem. Other forms of noise have practically no effect on generalization.