Learnable Bernoulli Dropout for Bayesian Deep Learning
Learnable Bernoulli Dropout for Bayesian Deep Learning
复制标题
DOI：
--
复制
发表时间：
2020-02
期刊：
影响因子：
--
通讯作者：
Shahin Boluki;Randy Ardywibowo;Siamak Zamani Dadaneh;Mingyuan Zhou;Xiaoning Qian
中科院分区：
文献类型：
--
作者：
Shahin Boluki;Randy Ardywibowo;Siamak Zamani Dadaneh;Mingyuan Zhou;Xiaoning Qian
在这项工作中，我们提出了可学习的伯努利辍学（LBD），这是一种新的模型无关的辍学方案，它将辍学率作为与其他模型参数共同优化的参数。通过对伯努利辍学的概率建模，我们的方法可以在深度模型中实现更稳健的预测和不确定性量化。特别是，当与变分自编码器（VAE）结合使用时，LBD实现了灵活的半隐式后验表示，从而产生了新的半隐式VAE （SIVAE）模型。我们使用augmentation - reinforcement - merge (ARM)，一种无偏和低方差梯度估计器，解决了关于dropout参数的训练优化问题。我们在一系列任务上的实验表明，与其他常用的退出方案相比，我们的方法具有优越的性能。总体而言，LBD可以提高图像分类和语义分割的准确性和不确定性估计。此外，使用SIVAE，我们可以在几个公共数据集的隐式反馈上实现最先进的协同过滤性能。
In this work, we propose learnable Bernoulli dropout (LBD), a new model-agnostic dropout scheme that considers the dropout rates as parameters jointly optimized with other model parameters. By probabilistic modeling of Bernoulli dropout, our method enables more robust prediction and uncertainty quantification in deep models. Especially, when combined with variational auto-encoders (VAEs), LBD enables flexible semi-implicit posterior representations, leading to new semi-implicit VAE~(SIVAE) models. We solve the optimization for training with respect to the dropout parameters using Augment-REINFORCE-Merge (ARM), an unbiased and low-variance gradient estimator. Our experiments on a range of tasks show the superior performance of our approach compared with other commonly used dropout schemes. Overall, LBD leads to improved accuracy and uncertainty estimates in image classification and semantic segmentation. Moreover, using SIVAE, we can achieve state-of-the-art performance on collaborative filtering for implicit feedback on several public datasets.