AutoShuffleNet: Learning Permutation Matrices via an Exact Lipschitz Continuous Penalty in Deep Convolutional Neural Networks
AutoShuffleNet: Learning Permutation Matrices via an Exact Lipschitz Continuous Penalty in Deep Convolutional Neural Networks
复制标题
DOI：
10.1145/3394486.3403103
复制
发表时间：
2019-01
期刊：
影响因子：
--
通讯作者：
J. Lyu;Shuai Zhang;Y. Qi;J. Xin
中科院分区：
文献类型：
--
作者：
J. Lyu;Shuai Zhang;Y. Qi;J. Xin
ShuffleNet是一个最先进的轻量级卷积神经网络架构。它的基本操作包括分组、信道卷积和信道变换。然而，信道变换是基于经验的人工设计。在数学上，洗牌是一个排列矩阵的乘法。在本文中，我们提出在网络训练中通过学习排列矩阵来实现信道洗牌的自动化。我们引入了一个精确的Lipschitz连续非凸惩罚，使它可以被纳入到随机梯度下降中，以高精度地近似排列。在训练结束时通过简单舍入得到精确排列，并用于推理。所得到的网络称为AutoShuffleNet，在保留ShuffleNet的推理成本的同时，对来自CIFAR-10、CIFAR-100和ImageNet的数据实现了更高的分类精度。此外，我们还通过实验发现，置换矩阵的标准凸松弛变成随机矩阵会导致性能不佳。从理论上证明了惩罚函数为零时恢复置换矩阵的准确性（误差界）。我们给出了通过图匹配和双层神经网络模型进行排列优化的例子，其中损失函数以封闭解析形式计算。在这些例子中，凸松弛未能捕捉到排列，而我们的惩罚却成功了。
ShuffleNet is a state-of-the-art light weight convolutional neural network architecture. Its basic operations include group, channel-wise convolution and channel shuffling. However, channel shuffling is manually designed on empirical grounds. Mathematically, shuffling is a multiplication by a permutation matrix. In this paper, we propose to automate channel shuffling by learning permutation matrices in network training. We introduce an exact Lipschitz continuous non-convex penalty so that it can be incorporated in the stochastic gradient descent to approximate permutation at high precision. Exact permutations are obtained by simple rounding at the end of training and are used in inference. The resulting network, referred to as AutoShuffleNet, achieved improved classification accuracies on data from CIFAR-10, CIFAR-100 and ImageNet while preserving the inference costs of ShuffleNet. In addition, we found experimentally that the standard convex relaxation of permutation matrices into stochastic matrices leads to poor performance. We prove theoretically the exactness (error bounds) in recovering permutation matrices when our penalty function is zero (very small). We present examples of permutation optimization through graph matching and two-layer neural network models where the loss functions are calculated in closed analytical form. In the examples, convex relaxation failed to capture permutations whereas our penalty succeeded.