Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections
Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections
复制标题
DOI：
--
复制
发表时间：
2021-02
期刊：
影响因子：
--
通讯作者：
A. Camuto;Xiaoyu Wang-;Lingjiong Zhu;Chris C. Holmes;M. Gürbüzbalaban;Umut Simsekli
中科院分区：
文献类型：
--
作者：
A. Camuto;Xiaoyu Wang-;Lingjiong Zhu;Chris C. Holmes;M. Gürbüzbalaban;Umut Simsekli
高斯噪声注入（GNI）是一系列用于训练神经网络的简单且广泛使用的正则化方法，其中在优化算法的每次迭代时将加性或乘性高斯噪声注入到网络激活中，通常选择为随机梯度下降（SGD）。在本文中，我们专注于所谓的“隐式效应”的GNIS，这是注入的噪声对SGD的动力学的影响。我们发现，这种效果会导致SGD梯度更新的非对称重尾噪声。为了模拟这种修改后的动力学，我们首先开发了一个Langevin随机微分方程，这是由一个一般家庭的非对称重尾噪声驱动。使用这个模型，我们正式证明国民总收入会引起“隐性偏差”，该偏差根据尾部的重量和不对称程度而变化。我们的实证结果证实，不同类型的GNI训练的神经网络是很好的建模所提出的动态和这些注入的隐式效果诱导的偏见，降低网络的性能。
Gaussian noise injections (GNIs) are a family of simple and widely-used regularisation methods for training neural networks, where one injects additive or multiplicative Gaussian noise to the network activations at every iteration of the optimisation algorithm, which is typically chosen as stochastic gradient descent (SGD). In this paper we focus on the so-called `implicit effect' of GNIs, which is the effect of the injected noise on the dynamics of SGD. We show that this effect induces an asymmetric heavy-tailed noise on SGD gradient updates. In order to model this modified dynamics, we first develop a Langevin-like stochastic differential equation that is driven by a general family of asymmetric heavy-tailed noise. Using this model we then formally prove that GNIs induce an `implicit bias', which varies depending on the heaviness of the tails and the level of asymmetry. Our empirical results confirm that different types of neural networks trained with GNIs are well-modelled by the proposed dynamics and that the implicit effect of these injections induces a bias that degrades the performance of networks.