Dropout vs. batch normalization: an empirical study of their impact to deep learning
Dropout vs. batch normalization: an empirical study of their impact to deep learning
复制标题
DOI：
10.1007/s11042-019-08453-9
复制
发表时间：
2020-05-01
影响因子：
3.6
通讯作者：
Marques, Oge
中科院分区：
文献类型：
--
作者：
Garbin, Christian;Zhu, Xingquan;Marques, Oge
过度拟合和训练时间过长是多层神经网络学习尤其是深度学习中面临的两个基本挑战。Dropout和批处理规范化是解决这些挑战的两种公认的方法。虽然这两种方法都有重叠的设计原则，但许多研究结果表明，它们在提高深度学习方面具有独特的优势。许多工具将这两种方法简化为一个简单的函数调用，允许灵活堆叠形成深度学习架构。虽然它们的使用指南是可用的，但不幸的是，没有定义良好的规则集或全面的研究来调查它们关于数据输入、网络配置、学习效率和准确性的问题。目前尚不清楚用户何时应该考虑使用dropout和/或批处理规范化，以及如何将它们组合（或替代使用）以实现优化的深度学习结果。在本文中，我们进行了一项实证研究，探讨dropout和批处理归一化对训练深度学习模型的影响。我们使用多层密集神经网络和卷积神经网络（CNN）作为深度学习模型，并混合dropout和批处理归一化来设计不同的架构，随后观察它们在训练和测试CPU时间、模型中的参数数量（作为模型大小的代理）和分类精度方面的性能。网络结构、dropout和批归一化之间的相互作用，使我们能够得出在深度学习中何时以及如何考虑dropout和批归一化的结论。该实证研究量化了使用dropout和批处理归一化时训练时间的增加，以及预测时间的增加（对于受限环境很重要，例如智能手机和低功耗物联网设备）。它表明，非自适应优化器（例如SGD）可以优于自适应优化器，但只需要花费大量的训练时间来执行超参数调优，而自适应优化器（例如RMSProp）在没有太多调优的情况下表现良好。最后，它表明，在cnn中使用dropout和批处理归一化应该谨慎和实验（当有疑问和实验时间短时，只使用批处理归一化）。
Overfitting and long training time are two fundamental challenges in multilayered neural network learning and deep learning in particular. Dropout and batch normalization are two well-recognized approaches to tackle these challenges. While both approaches share overlapping design principles, numerous research results have shown that they have unique strengths to improve deep learning. Many tools simplify these two approaches as a simple function call, allowing flexible stacking to form deep learning architectures. Although their usage guidelines are available, unfortunately no well-defined set of rules or comprehensive studies to investigate them concerning data input, network configurations, learning efficiency, and accuracy. It is not clear when users should consider using dropout and/or batch normalization, and how they should be combined (or used alternatively) to achieve optimized deep learning outcomes. In this paper we conduct an empirical study to investigate the effect of dropout and batch normalization on training deep learning models. We use multilayered dense neural networks and convolutional neural networks (CNN) as the deep learning models, and mix dropout and batch normalization to design different architectures and subsequently observe their performance in terms of training and test CPU time, number of parameters in the model (as a proxy for model size), and classification accuracy. The interplay between network structures, dropout, and batch normalization, allow us to conclude when and how dropout and batch normalization should be considered in deep learning. The empirical study quantified the increase in training time when dropout and batch normalization are used, as well as the increase in prediction time (important for constrained environments, such as smartphones and low-powered IoT devices). It showed that a non-adaptive optimizer (e.g. SGD) can outperform adaptive optimizers, but only at the cost of a significant amount of training times to perform hyperparameter tuning, while an adaptive optimizer (e.g. RMSProp) performs well without much tuning. Finally, it showed that dropout and batch normalization should be used in CNNs only with caution and experimentation (when in doubt and short on time to experiment, use only batch normalization).