A Novel Rank Selection Scheme in Tensor Ring Decomposition Based on Reinforcement Learning for Deep Neural Networks
A Novel Rank Selection Scheme in Tensor Ring Decomposition Based on Reinforcement Learning for Deep Neural Networks
复制标题
一种基于深度神经网络强化学习的张量环分解秩选择方案
DOI：
--
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
Sid Ying
中科院分区：
文献类型：
--
作者：
Zhiyu Cheng;Baopu Li;Yanwen Fan;Sid Ying
张量分解已被证明是解决信号处理和机器学习中的许多问题的有效方法。近年来，张量分解在压缩深度神经网络方面发现了它的优势。在深度神经网络的许多应用中，减少参数数量和计算量是加快网络部署推理速度的关键。现代深度神经网络由具有多数组权重的多层组成，其中张量分解是执行压缩的自然方式。它是通过分解卷积层或具有指定张量秩（例如规范秩，张量列秩）的全连接层中的权张量来实现的。传统的压缩深度神经网络的张量分解方法需要人工选择秩，这需要繁琐的人力来调整性能。为了克服这一问题，我们提出了一种新的秩选择方案，该方案受强化学习的启发，在最近研究的张量环分解中自动选择每个卷积层的秩。实验结果证实，我们基于学习的排名选择在许多基准数据集上显著优于手工制作的排名选择启发式，目的是有效压缩深度神经网络，同时保持相当的准确性。
Tensor decomposition has been proved to be effective for solving many problems in signal processing and machine learning[1]. Recently, tensor decomposition finds its advantage for compressing deep neural networks. In many applications of deep neural networks, it is critical to reduce the number of parameters and computation workload to accelerate inference speed in deployment of the network. Modern deep neural network consists of multiple layers with multi-array weights where tensor decomposition is a natural way to perform compression. It is achieved by decomposing the weight tensors in convolutional layers or fully-connected layers with specified tensor ranks (e.g. canonical ranks, tensor train ranks). Conventional tensor decomposition in compressing deep neural networks selects the ranks manually that requires tedious human efforts to finetune the performance. To overcome this issue, we propose a novel rank selection scheme, which is inspired by reinforcement learning, to automatically select ranks in recently studied tensor ring decomposition in each convolutional layer. Experimental results validate that our learning based rank selection significantly outperforms hand-crafted rank selection heuristics on a number of benchmark datasets, for the purpose of effectively compressing deep neural networks while maintaining comparable accuracy.