A novel structured sparse fully connected layer in convolutional neural networks
A novel structured sparse fully connected layer in convolutional neural networks
复制标题
DOI：
10.1002/cpe.6213
复制
发表时间：
2021-01-31
影响因子：
2
通讯作者：
Tabaru, Tsuguchika
中科院分区：
文献类型：
--
作者：
Matsumura, Naoki;Ito, Yasuaki;Tabaru, Tsuguchika
卷积神经网络（CNN）是支撑人工智能技术快速发展的因素之一。然而，随着网络能力的增加，网络的规模也变得更大。到目前为止，与减少网络规模相关的几项工作已经得到解决。在许多情况下，这些方法会产生非结构化网络，从而阻碍高效的并行计算。为了避免这个问题，我们在 CNN 中提出了一种新颖的结构化稀疏全连接层（FCL）。我们提出的方法的目的是减少 FCL 中占据大部分网络参数的网络参数的数量。与流行的 CNN（例如 VGG-16）中使用的通用 FCL 不同，所提出的方法减少了最后一个卷积层和第一个 FCL 之间的连接。此外，我们还展示了使用 cuBLAS 在 GPU 上实现所提出的稀疏 FCL 的方法。对于 ILSVRC-2012 数据集，所提出的方法实现了 21.3 倍的压缩，对于 VGG-16，top-1 精度降低了 0.68%，top-5 精度降低了 0.31%。与非压缩 FCL 相比，所提出的 FCL 的实现在前向和后向传播方面分别实现了 14.97 和 16.67 的加速因子。
Convolutional Neural Networks (CNNs) are one of the factors supporting the rapid development of artificial intelligent techniques. However, as the ability of the network increases, the size of the network becomes larger. Thus far, several works related to reduction of the network size have been tackled. In many cases, these approaches produce an unstructured network which prevents efficient parallel computation. To avoid this problem, we propose a novel structured sparse fully connected layer (FCL) in the CNNs. The aim of our proposed approach is reduction of the number of network parameters in the FCLs which occupy a large part of network parameters. Unlike the general FCLs used in the popular CNNs such as VGG-16, the proposed approach reduces the connection between the last convolutional layer and the first FCL. In addition, we show an implementation for the proposed sparse FCLs on the GPU using cuBLAS. As a result for ILSVRC-2012 dataset, the proposed approach achieves a 21.3 times compression with 0.68% top-1 accuracy and 0.31% top-5 accuracy decreases for VGG-16. The implementation of the proposed FCLs achieves speed-up factor 14.97 and 16.67 for forward and backward propagation compared to that for the noncompressed FCLs, respectively.