Training Faster by Separating Modes of Variation in Batch-Normalized Models
Training Faster by Separating Modes of Variation in Batch-Normalized Models
复制标题
DOI：
10.1109/tpami.2019.2895781
复制
发表时间：
2020-06-01
影响因子：
23.6
通讯作者：
Shah, Mubarak
中科院分区：
文献类型：
--
作者：
Kalayeh, Mahdi M.;Shah, Mubarak
批量归一化 (BN) 对于有效训练最先进的深度卷积神经网络 (CNN) 至关重要。它使用每个小批量的统计数据对训练期间的层输出进行归一化。 BN 通过允许安全地利用大学习率来加速训练过程，并减少仔细初始化参数的需要。在这项工作中，我们从生成概率模型产生的 Fisher 核的角度研究 BN。我们证明，假设小批量中的样本来自相同的概率密度函数，则 BN 与高斯分布的 Fisher 向量相同。这意味着批量归一化变换可以用内核来解释，这些内核自然地从概率密度函数中出现，该函数对底层数据分布的生成过程进行建模。因此，它保证了批量归一化的小批量具有更高的辨别力。然而，考虑到 CNN 架构中采用的非线性校正，层输出的分布表现出不对称特征。因此，为了让 BN 充分受益于上述特性，我们建议不使用一种高斯密度，而是使用多种高斯密度的混合来近似底层数据分布。推导高斯混合模型 (GMM) 的 Fisher 向量表明，可以通过对解开的子群体的统计数据进行独立标准化来改进批量标准化。我们将我们提出的批量归一化的软分段版本称为混合归一化（MN）。通过在 CIFAR-10 和 CIFAR-100 上进行的大量实验，使用 5 层深度 CNN 和现代 Inception-V3 架构，我们表明混合归一化可减少所需的梯度更新次数，以在各种训练场景中将批量归一化模型的最大测试精度降低 $\sim 31\%-47\%$similar 至 31%-47%。在 48 层深度的 Inception-V3 架构中，即使用 MN 替换几个 BN 模块也足以不仅获得可观的训练加速，而且能获得更好的最终测试精度。我们表明，类似的观察结果对于 40 层和 100 层深 DenseNet 架构也有效。我们通过评估混合归一化在生成对抗网络（GAN）中的应用来补充我们的研究，其中“模式崩溃”阻碍了训练过程。我们仅用我们提出的混合归一化替换生成器中的一些批量归一化层。我们在 CIFAR-10 上使用深度卷积 GAN (DCGAN) 的实验表明，混合归一化 DCGAN 不仅提供了接近 58% 的加速 $\sim 58\%$，而且与批量归一化对应物的 37.56 相比，“Frechet 起始距离”(FID) 达到了更低（更好）的 33.35。
Batch Normalization (BN) is essential to effectively train state-of-the-art deep Convolutional Neural Networks (CNN). It normalizes the layer outputs during training using the statistics of each mini-batch. BN accelerates training procedure by allowing to safely utilize large learning rates and alleviates the need for careful initialization of the parameters. In this work, we study BN from the viewpoint of Fisher kernels that arise from generative probability models. We show that assuming samples within a mini-batch are from the same probability density function, then BN is identical to the Fisher vector of a Gaussian distribution. That means batch normalizing transform can be explained in terms of kernels that naturally emerge from the probability density function that models the generative process of the underlying data distribution. Consequently, it promises higher discrimination power for the batch-normalized mini-batch. However, given the rectifying non-linearities employed in CNN architectures, distribution of the layer outputs show an asymmetric characteristic. Therefore, in order for BN to fully benefit from the aforementioned properties, we propose approximating underlying data distribution not with one, but a mixture of Gaussian densities. Deriving Fisher vector for a Gaussian Mixture Model (GMM), reveals that batch normalization can be improved by independently normalizing with respect to the statistics of disentangled sub-populations. We refer to our proposed soft piecewise version of batch normalization as Mixture Normalization (MN). Through extensive set of experiments on CIFAR-10 and CIFAR-100, using both a 5-layers deep CNN and modern Inception-V3 architecture, we show that mixture normalization reduces required number of gradient updates to reach the maximum test accuracy of the batch-normalized model by $\sim 31\%-47\%$similar to 31%-47% across a variety of training scenarios. Replacing even a few BN modules with MN in the 48-layers deep Inception-V3 architecture is sufficient to not only obtain considerable training acceleration but also better final test accuracy. We show that similar observations are valid for 40 and 100-layers deep DenseNet architectures as well. We complement our study by evaluating the application of mixture normalization to the Generative Adversarial Networks (GANs), where "mode collapse" hinders the training process. We solely replace a few batch normalization layers in the generator with our proposed mixture normalization. Our experiments using Deep Convolutional GAN (DCGAN) on CIFAR-10 show that mixture-normalized DCGAN not only provides an acceleration of $\sim 58\%$similar to 58% but also reaches lower (better) "Frechet Inception Distance" (FID) of 33.35 compared to 37.56 of its batch-normalized counterpart.