An Adaptive Empirical Bayesian Method for Sparse Deep Learning
An Adaptive Empirical Bayesian Method for Sparse Deep Learning
复制标题
DOI：
--
复制
发表时间：
2019-10
期刊：
影响因子：
--
通讯作者：
Wei Deng;Xiao Zhang;F. Liang;Guang Lin
中科院分区：
文献类型：
--
作者：
Wei Deng;Xiao Zhang;F. Liang;Guang Lin
我们提出了一种新的用于稀疏深度学习的自适应经验贝叶斯（AEB）方法，其中通过一类自适应spike- slab先验来确保稀疏性。该方法使用随机梯度马尔可夫链蒙特卡罗（MCMC）从自适应分层后验分布中交替采样，并使用随机逼近（SA）平滑优化超参数。进一步证明了该方法在温和条件下对渐近正确分布的收敛性。经验应用表明，该方法在使用浅卷积神经网络（CNN）的MNIST和时尚MNIST上具有最先进的性能，并且在使用残差网络的CIFAR10上具有最先进的压缩性能。该方法还提高了对对抗性攻击的抵抗力。
We propose a novel adaptive empirical Bayesian (AEB) method for sparse deep learning, where the sparsity is ensured via a class of self-adaptive spike-and-slab priors. The proposed method works by alternatively sampling from an adaptive hierarchical posterior distribution using stochastic gradient Markov Chain Monte Carlo (MCMC) and smoothly optimizing the hyperparameters using stochastic approximation (SA). We further prove the convergence of the proposed method to the asymptotically correct distribution under mild conditions. Empirical applications of the proposed method lead to the state-of-the-art performance on MNIST and Fashion MNIST with shallow convolutional neural networks (CNN) and the state-of-the-art compression performance on CIFAR10 with Residual Networks. The proposed method also improves resistance to adversarial attacks.