GRIM: A General, Real-Time Deep Learning Inference Framework for Mobile Devices Based on Fine-Grained Structured Weight Sparsity
GRIM: A General, Real-Time Deep Learning Inference Framework for Mobile Devices Based on Fine-Grained Structured Weight Sparsity
复制标题
DOI：
10.1109/tpami.2021.3089687
复制
发表时间：
2021-06
影响因子：
23.6
通讯作者：
Wei Niu;Zhengang;Xiaolong Ma;Peiyan Dong;Gang Zhou;Xuehai Qian;Xue Lin;Yanzhi Wang;Bin Ren
中科院分区：
文献类型：
--
作者：
Wei Niu;Zhengang;Xiaolong Ma;Peiyan Dong;Gang Zhou;Xuehai Qian;Xue Lin;Yanzhi Wang;Bin Ren
在移动的设备上实现实时深度神经网络（DNN）推理是有吸引力但具有挑战性的，因为即使是功能强大的现代移动的设备在执行大规模DNN时也被认为是“资源受限的”。它需要通过权重修剪进行稀疏模型推理，即，DNN权重稀疏性，并且期望设计一种新的DNN权重稀疏性方案，其可以促进在移动的设备上的实时推断，同时保持高稀疏模型精度。本文设计了一种新的移动的推理加速框架GRIM，该框架通用于卷积神经网络（CNN）和递归神经网络（RNN），并实现了实时执行和高准确性，利用细粒度结构化稀疏模型推理和编译器优化移动。首先，我们提出了一个新的细粒度结构稀疏方案，通过基于块的列行（BCR）修剪。基于这种新的细粒度结构化稀疏性，我们的GRIM框架由两部分组成：（a）实时移动的推理的编译器优化和代码生成;（B）BCR剪枝优化，用于确定剪枝超参数和执行权重剪枝。我们将GRIM与Alibaba MNN，TVM，TensorFlow-Lite，基于CSR，PatDNN和ESE（RNN的代表性FPGA推理加速框架）的稀疏实现进行了比较，并实现了高达14.08 × 14.08 ×的加速比。
It is appealing but challenging to achieve real-time deep neural network (DNN) inference on mobile devices, because even the powerful modern mobile devices are considered as “resource-constrained” when executing large-scale DNNs. It necessitates the sparse model inference via weight pruning, i.e., DNN weight sparsity, and it is desirable to design a new DNN weight sparsity scheme that can facilitate real-time inference on mobile devices while preserving a high sparse model accuracy. This paper designs a novel mobile inference acceleration framework GRIM that is General to both convolutional neural networks (CNNs) and recurrent neural networks (RNNs) and that achieves Real-time execution and high accuracy, leveraging fine-grained structured sparse model Inference and compiler optimizations for Mobiles. We start by proposing a new fine-grained structured sparsity scheme through the Block-based Column-Row (BCR) pruning. Based on this new fine-grained structured sparsity, our GRIM framework consists of two parts: (a) the compiler optimization and code generation for real-time mobile inference; and (b) the BCR pruning optimizations for determining pruning hyperparameters and performing weight pruning. We compare GRIM with Alibaba MNN, TVM, TensorFlow-Lite, a sparse implementation based on CSR, PatDNN, and ESE (a representative FPGA inference acceleration framework for RNNs), and achieve up to $14.08\times$14.08× speedup.