Group Normalization
Group Normalization
复制标题
DOI：
10.1109/cstic.2018.8369274
复制
发表时间：
2020-03-01
影响因子：
19.5
通讯作者：
He, Kaiming
中科院分区：
文献类型：
--
作者：
Wu, Yuxin;He, Kaiming
批量归一化（BN）是深度学习发展中的一项里程碑式的技术，它使各种网络能够进行训练。然而，规范化沿着批量尺寸引入的问题-BN的错误迅速增加时，批量大小变得更小，造成不准确的批量统计估计。这限制了BN在训练更大模型和将特征转移到计算机视觉任务（包括检测、分割和视频）中的使用，这些任务需要小批量处理，受到内存消耗的限制。在本文中，我们提出了群归一化（GN）作为BN的一个简单的替代方案。GN将通道划分为组，并在每个组内计算归一化的均值和方差。GN的计算与批量大小无关，其精度在很大的批量大小范围内都是稳定的。在ImageNet中训练的ResNet-50上，当使用批量大小为2时，GN的误差比BN的误差低10.6%;当使用典型的批量大小时，GN与BN相当，并且优于其他归一化变体。此外，GN可以自然地从预训练转移到微调。GN在COCO（）中的对象检测和分割以及Kinetics中的视频分类方面优于基于BN的同行，表明GN可以在各种任务中有效地取代强大的BN。GN可以在现代库中通过几行代码轻松实现。
Batch Normalization (BN) is a milestone technique in the development of deep learning, enabling various networks to train. However, normalizing along the batch dimension introduces problems-BN's error increases rapidly when the batch size becomes smaller, caused by inaccurate batch statistics estimation. This limits BN's usage for training larger models and transferring features to computer vision tasks including detection, segmentation, and video, which require small batches constrained by memory consumption. In this paper, we present Group Normalization (GN) as a simple alternative to BN. GN divides the channels into groups and computes within each group the mean and variance for normalization. GN's computation is independent of batch sizes, and its accuracy is stable in a wide range of batch sizes. On ResNet-50 trained in ImageNet, GN has 10.6% lower error than its BN counterpart when using a batch size of 2; when using typical batch sizes, GN is comparably good with BN and outperforms other normalization variants. Moreover, GN can be naturally transferred from pre-training to fine-tuning. GN can outperform its BN-based counterparts for object detection and segmentation in COCO (), and for video classification in Kinetics, showing that GN can effectively replace the powerful BN in a variety of tasks. GN can be easily implemented by a few lines of code in modern libraries.