Revisit Batch Normalization: New Understanding and Refinement via Composition Optimization
Revisit Batch Normalization: New Understanding and Refinement via Composition Optimization
复制标题
重新审视批量归一化：通过成分优化获得新的理解和细化
DOI：
--
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
Ji Liu
中科院分区：
文献类型：
--
作者：
Xiangru Lian;Ji Liu
分批归一化（BN）已在深度学习中广泛使用，以实现更快的训练过程和更好的模型。但是，BN是否工作在很大程度上取决于培训期间批处理的构建方式，并且如果批处理上的统计信息与整个数据集中的统计数据不接近，则可能不会收敛到所需的解决方案。在本文中，我们通过提供与BN相关的明确目标函数来从优化的角度理解BN。这种明确的目标函数表明：1）bn而不是成为一种新的优化算法或技巧，而是创建一个不同的目标函数，而不是我们的常识； 2）为什么BN在某些情况下可能无法正常工作。
然后，我们根据称为完全归一化（FN）的组成优化技术提出了BN的改进，以减轻当批次未理想构造时BN的问题。 FN的收敛分析和经验研究也包括在本文中。
Batch Normalization (BN) has been used extensively in deep learning to achieve faster training process and better resulting models. However, whether BN works strongly depends on how the batches are constructed during training, and it may not converge to a desired solution if the statistics on the batch are not close to the statistics over the whole dataset. In this paper, we try to understand BN from an optimization perspective by providing an explicit objective function associated with BN. This explicit objective function reveals that: 1) BN, rather than being a new optimization algorithm or trick, is creating a different objective function instead of the one in our common sense; and 2) why BN may not work well in some scenarios.
We then propose a refinement of BN based on the compositional optimization technique called Full Normalization (FN) to alleviate the issues of BN when the batches are not constructed ideally. The convergence analysis and empirical study for FN are also included in this paper.
DOI：
10.1609/aaai.v32i1.11795
发表时间：
2017-11
期刊：
ArXiv
影响因子：
--
作者：
Zhouyuan Huo;Bin Gu;Heng Huang
通讯作者：
Zhouyuan Huo;Bin Gu;Heng Huang