Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training-猫眼课题宝

DOI：

--

发表时间：

2021-03

期刊：

--

影响因子：

--

通讯作者：

中科院分区：

其他

文献类型：

--

作者：

Sheng Liu;Xiao Li;Yuexiang Zhai;Chong You;Zhihui Zhu;C. Fernandez‐Granda;Qing Qu

归一化技术已经成为现代卷积神经网络（ConvNets）的基本组成部分。特别是，最近的许多工作表明，提高权重的正交性有助于训练深度模型并提高鲁棒性。对于ConvNets，大多数现有方法都基于对卷积核进行连接或压平而得出的权重矩阵进行惩罚或归一化。这些方法通常会破坏或忽略内核的良性卷积结构;因此，它们对于深度ConvNets来说通常是昂贵或不切实际的。相比之下，我们引入了一种简单有效的“卷积归一化”（ConvNorm）方法，该方法可以充分利用傅立叶域中的卷积结构，并作为一个简单的即插即用模块，方便地集成到任何ConvNet中。我们的方法的灵感来自于最近的工作预处理方法的卷积稀疏编码，可以有效地促进每一层的通道的等距。此外，我们还证明了我们的ConvNorm可以降低权重矩阵的逐层谱范数，从而提高网络的Lipschitz性，从而使深度ConvNets的训练更容易，鲁棒性更高。应用于噪声污染和生成对抗网络（GAN）下的分类，我们证明了ConvNorm提高了ResNet等常见ConvNets的鲁棒性和GAN的性能。我们通过CIFAR和ImageNet上的数值实验验证了我们的发现。

Normalization techniques have become a basic component in modern convolutional neural networks (ConvNets). In particular, many recent works demonstrate that promoting the orthogonality of the weights helps train deep models and improve robustness. For ConvNets, most existing methods are based on penalizing or normalizing weight matrices derived from concatenating or flattening the convolutional kernels. These methods often destroy or ignore the benign convolutional structure of the kernels; therefore, they are often expensive or impractical for deep ConvNets. In contrast, we introduce a simple and efficient"Convolutional Normalization"(ConvNorm) method that can fully exploit the convolutional structure in the Fourier domain and serve as a simple plug-and-play module to be conveniently incorporated into any ConvNets. Our method is inspired by recent work on preconditioning methods for convolutional sparse coding and can effectively promote each layer's channel-wise isometry. Furthermore, we show that our ConvNorm can reduce the layerwise spectral norm of the weight matrices and hence improve the Lipschitzness of the network, leading to easier training and improved robustness for deep ConvNets. Applied to classification under noise corruptions and generative adversarial network (GAN), we show that the ConvNorm improves the robustness of common ConvNets such as ResNet and the performance of GAN. We verify our findings via numerical experiments on CIFAR and ImageNet.