FracTrain: Fractionally Squeezing Bit Savings Both Temporally and Spatially for Efficient DNN Training
FracTrain: Fractionally Squeezing Bit Savings Both Temporally and Spatially for Efficient DNN Training
复制标题
DOI：
--
复制
发表时间：
2020-12
期刊：
影响因子：
--
通讯作者：
Y. Fu;Haoran You;Yang Zhao;Yue Wang;Chaojian Li;K. Gopalakrishnan;Zhangyang Wang;Yingyan Lin
中科院分区：
文献类型：
--
作者：
Y. Fu;Haoran You;Yang Zhao;Yue Wang;Chaojian Li;K. Gopalakrishnan;Zhangyang Wang;Yingyan Lin
深度神经网络（DNN）的最新突破推动了对具有现场学习功能的智能边缘设备的巨大需求，而由于边缘可用资源有限以及最先进（SOTA）DNN所需的大量培训成本，此类系统的实际实现仍然是一个挑战。由于降低精度是提高训练时间/能量效率的最有效方法之一，因此人们对低精度DNN训练的兴趣越来越大。在本文中，我们从正交方向探索：如何从最冗余的比特级别，逐步沿着训练轨迹和动态地每个输入，逐步挤出更多的训练成本节省。具体来说，我们提出了FracTrain，它集成了（i）渐进分数量化，它逐渐增加激活，权重和梯度的精度，直到最后的训练阶段才达到SOTA静态量化DNN训练的精度，以及（ii）动态分数量化，它以输入自适应的方式为每层的激活和梯度分配精度，用于仅“部分地“更新层参数。广泛的模拟和消融研究（六个模型，四个数据集和三个训练设置，包括标准，自适应和微调）验证了FracTrain在降低DNN训练的计算成本和硬件量化能量/延迟方面的有效性，同时实现了相当或更好的准确性（-0.12%~+1.87%）。例如，当在CIFAR-10上训练ResNet-74时，与最佳SOTA基线相比，FracTrain分别实现了77.6%和53.5%的计算成本和训练延迟节省，同时实现了相当的（-0.07%）准确度。我们的代码可在https://github.com/RICE-EIC/FracTrain上获得。
Recent breakthroughs in deep neural networks (DNNs) have fueled a tremendous demand for intelligent edge devices featuring on-site learning, while the practical realization of such systems remains a challenge due to the limited resources available at the edge and the required massive training costs for state-of-the-art (SOTA) DNNs. As reducing precision is one of the most effective knobs for boosting training time/energy efficiency, there has been a growing interest in low-precision DNN training. In this paper, we explore from an orthogonal direction: how to fractionally squeeze out more training cost savings from the most redundant bit level, progressively along the training trajectory and dynamically per input. Specifically, we propose FracTrain that integrates (i) progressive fractional quantization which gradually increases the precision of activations, weights, and gradients that will not reach the precision of SOTA static quantized DNN training until the final training stage, and (ii) dynamic fractional quantization which assigns precisions to both the activations and gradients of each layer in an input-adaptive manner, for only"fractionally"updating layer parameters. Extensive simulations and ablation studies (six models, four datasets, and three training settings including standard, adaptation, and fine-tuning) validate the effectiveness of FracTrain in reducing computational cost and hardware-quantified energy/latency of DNN training while achieving a comparable or better (-0.12%~+1.87%) accuracy. For example, when training ResNet-74 on CIFAR-10, FracTrain achieves 77.6% and 53.5% computational cost and training latency savings, respectively, compared with the best SOTA baseline, while achieving a comparable (-0.07%) accuracy. Our codes are available at: https://github.com/RICE-EIC/FracTrain.