Dynamic Neural Network to Enable Run-Time Trade-off between Accuracy and Latency
Dynamic Neural Network to Enable Run-Time Trade-off between Accuracy and Latency
复制标题
DOI：
10.1145/3394885.3431628
复制
发表时间：
2021-01
期刊：
影响因子：
--
通讯作者：
Li Yang;Deliang Fan
中科院分区：
文献类型：
--
作者：
Li Yang;Deliang Fan
为了将强大的深度神经网络（DNN）部署到智能但资源有限的物联网设备中，人们提出了许多先前的工作来压缩DNN，以减少网络大小和计算复杂性，同时精度下降可以忽略不计，例如权重量化、网络剪枝、卷积分解等。然而，通过利用传统的DNN压缩方法，从相对较大的背景模型生成更小但固定的网络，以实现资源有限的硬件加速。然而，这种优化缺乏实时调整其结构以适应动态计算硬件资源分配和工作负载的能力。在本文中，我们主要回顾了我们之前的两项工作[13]、[15]来应对这一挑战，讨论如何通过均匀或非均匀子网生成方法构建动态 DNN。此外，为了生成多个非均匀子网，[15]需要单独对每个子网完全重新训练背景模型，称为多路径方法。为了降低训练成本，在这项工作中，我们进一步提出了一种单路径子网生成方法，可以在一轮训练中对不同时期的多个子网进行采样。构建的动态DNN由多个子网组成，提供了根据硬件资源和环境要求在运行时权衡推理精度和延迟的能力。最后，我们在 CIFAR-10 和 ImageNet 数据集上研究了具有不同子网生成方法的动态 DNN。我们还介绍了 GPU 和 CPU 上的准确性和延迟的运行时调整。
To deploy powerful deep neural network (DNN) into smart, but resource limited IoT devices, many prior works have been proposed to compress DNN to reduce the network size and computation complexity with negligible accuracy degradation, such as weight quantization, network pruning, convolution decomposition, etc. However, by utilizing conventional DNN compression methods, a smaller, but fixed, network is generated from a relative large background model to achieve resource limited hardware acceleration. However, such optimization lacks the ability to adjust its structure in real-time to adapt for a dynamic computing hardware resource allocation and workloads. In this paper, we mainly review our two prior works [13], [15] to tackle this challenge, discussing how to construct a dynamic DNN by means of either uniform or non-uniform sub-nets generation methods. Moreover, to generate multiple non-uniform sub-nets, [15] needs to fully retrain the background model for each sub-net individually, named as multi-path method. To reduce the training cost, in this work, we further propose a single-path sub-nets generation method that can sample multiple sub-nets in different epochs within one training round. The constructed dynamic DNN, consisting of multiple sub-nets, provides the ability to run-time trade-off the inference accuracy and latency according to hardware resources and environment requirements. In the end, we study the the dynamic DNNs with different sub-nets generation methods on both CIFAR-10 and ImageNet dataset. We also present the run-time tuning of accuracy and latency on both GPU and CPU.