Convolutional neural networks with low-rank regularization
Convolutional neural networks with low-rank regularization
复制标题
DOI：
--
复制
发表时间：
2015-11
期刊：
影响因子：
--
通讯作者：
Cheng Tai;Tong Xiao;Xiaogang Wang;E. Weinan
中科院分区：
文献类型：
--
作者：
Cheng Tai;Tong Xiao;Xiaogang Wang;E. Weinan
大型CNN在各种计算机视觉应用中提供了令人印象深刻的性能。但是存储和计算需求使得在移动的设备上部署这些模型成为问题。最近，张量分解已被用于加速CNN。在本文中，我们进一步发展了张量分解技术。我们提出了一种新的算法来计算低秩张量分解，以消除卷积核中的冗余。该算法找到了分解的精确全局优化器，比迭代方法更有效。基于分解，我们进一步提出了一种从头开始训练低秩约束CNN的新方法。有趣的是，虽然实现了显着的加速，但有时低秩约束CNN的性能明显优于非约束CNN。在CIFAR-10数据集上，提出的低秩NIN模型达到了91.31\%$的准确性（无需数据增强），这也比最先进的结果有所改进。我们在CIFAR-10和ILSVRC 12数据集上评估了所提出的方法，用于各种现代CNN，包括AlexNet，NIN，VGG和GoogleNet。例如，VGG-16的转发时间减少了一半，而性能仍然相当。经验上的成功表明，低秩张量分解可以成为加速大型CNN的非常有用的工具。
Large CNNs have delivered impressive performance in various computer vision applications. But the storage and computation requirements make it problematic for deploying these models on mobile devices. Recently, tensor decompositions have been used for speeding up CNNs. In this paper, we further develop the tensor decomposition technique. We propose a new algorithm for computing the low-rank tensor decomposition for removing the redundancy in the convolution kernels. The algorithm finds the exact global optimizer of the decomposition and is more effective than iterative methods. Based on the decomposition, we further propose a new method for training low-rank constrained CNNs from scratch. Interestingly, while achieving a significant speedup, sometimes the low-rank constrained CNNs delivers significantly better performance than their non-constrained counterparts. On the CIFAR-10 dataset, the proposed low-rank NIN model achieves $91.31\%$ accuracy (without data augmentation), which also improves upon state-of-the-art result. We evaluated the proposed method on CIFAR-10 and ILSVRC12 datasets for a variety of modern CNNs, including AlexNet, NIN, VGG and GoogleNet with success. For example, the forward time of VGG-16 is reduced by half while the performance is still comparable. Empirical success suggests that low-rank tensor decompositions can be a very useful tool for speeding up large CNNs.