NeuLens: spatial-based dynamic acceleration of convolutional neural networks on edge
NeuLens: spatial-based dynamic acceleration of convolutional neural networks on edge
复制标题
DOI：
10.1145/3495243.3560528
复制
发表时间：
2022-10
期刊：
影响因子：
--
通讯作者：
Xueyu Hou;Yongjie Guan;Tao Han
中科院分区：
文献类型：
--
作者：
Xueyu Hou;Yongjie Guan;Tao Han
卷积神经网络（CNN）在当今的移动的和边缘计算系统中发挥着重要作用，用于对象分类和检测等基于视觉的任务。然而，关于CNN加速的最先进的方法被困在通用计算平台上的有限的实际延迟加速或具有严重准确性损失的延迟加速。在本文中，我们提出了一个基于空间的动态CNN加速框架NeuLens，用于移动的和边缘平台。特别是，我们设计了一种新的动态推理机制，组装区域感知卷积（ARAC）超网，剥离CNN模型内部的冗余操作尽可能多的空间冗余和通道切片的基础上。在ARAC超网中，CNN推理流被分成多个独立的微流，每个微流的计算成本可以根据其平铺输入内容和应用需求自主调整。这些微流可以作为单个模型加载到GPU等硬件中。因此，它的操作减少可以很好地转化为延迟加速，并与硬件级加速兼容。此外，通过识别图像上的关键区域并在具有大微流的原始分辨率下处理它们，可以很好地保持推理精度。根据我们的评估，NeuLens优于基线方法，在相同精度的情况下，延迟减少高达58%，在相同延迟/内存限制下，精度提高高达67.9%。
Convolutional neural networks (CNNs) play an important role in today's mobile and edge computing systems for vision-based tasks like object classification and detection. However, state-of-the-art methods on CNN acceleration are trapped in either limited practical latency speed-up on general computing platforms or latency speed-up with severe accuracy loss. In this paper, we propose a spatial-based dynamic CNN acceleration framework, NeuLens, for mobile and edge platforms. Specially, we design a novel dynamic inference mechanism, assemble region-aware convolution (ARAC) supernet, that peels off redundant operations inside CNN models as many as possible based on spatial redundancy and channel slicing. In ARAC supernet, the CNN inference flow is split into multiple independent micro-flows, and the computational cost of each can be autonomously adjusted based on its tiled-input content and application requirements. These micro-flows can be loaded into hardware like GPUs as single models. Consequently, its operation reduction can be well translated into latency speed-up and is compatible with hardware-level accelerations. Moreover, the inference accuracy can be well preserved by identifying critical regions on images and processing them in the original resolution with large micro-flow. Based on our evaluation, NeuLens outperforms baseline methods by up to 58% latency reduction with the same accuracy and by up to 67.9% accuracy improvement under the same latency/memory constraints.