LDP: Learnable Dynamic Precision for Efficient Deep Neural Network Training and Inference
LDP: Learnable Dynamic Precision for Efficient Deep Neural Network Training and Inference
复制标题
DOI：
10.48550/arxiv.2203.07713
复制
发表时间：
2022-03
期刊：
影响因子：
--
通讯作者：
Zhongzhi Yu;Y. Fu;Shang Wu;Mengquan Li;Haoran You;Yingyan Lin
中科院分区：
文献类型：
--
作者：
Zhongzhi Yu;Y. Fu;Shang Wu;Mengquan Li;Haoran You;Yingyan Lin
低精度深度神经网络（DNN）训练是提高DNN训练效率的最有效技术之一，因为它从最精细的比特级别削减了训练成本。虽然现有的工作大多在整个训练过程中固定模型精度，但一些开创性的工作表明，动态精度调度有助于DNN收敛到更好的精度，同时导致比静态精度训练更低的训练成本。然而，现有的动态低精度训练方法依赖于手动设计的精度时间表来实现有利的效率和精度权衡，限制了它们更全面的实际应用和可实现的性能。为此，我们提出了LDP，这是一个可学习的动态精度DNN训练框架，可以在训练过程中自动学习时间和空间动态精度计划，以实现最佳精度和效率权衡。值得注意的是，LDP训练的DNN在推理过程中本质上是高效的。此外，我们可视化了LDP训练的DNN在不同任务上的时间和空间精度调度和分布，以更好地了解相应的DNN在不同训练阶段和DNN层的特性，在训练期间和训练之后，为促进进一步的创新提供见解。广泛的实验和消融研究（七个网络，五个数据集和三个任务）表明，所提出的LDP在训练效率和实现的准确性权衡方面始终优于最先进的（SOTA）低精度DNN训练技术。例如，除了具有自动化的优势外，与最好的SOTA方法相比，我们的LDP在CIFAR-10上训练ResNet-20时，准确率提高了0.31%，计算成本降低了39.1%。
Low precision deep neural network (DNN) training is one of the most effective techniques for boosting DNNs' training efficiency, as it trims down the training cost from the finest bit level. While existing works mostly fix the model precision during the whole training process, a few pioneering works have shown that dynamic precision schedules help DNNs converge to a better accuracy while leading to a lower training cost than their static precision training counterparts. However, existing dynamic low precision training methods rely on manually designed precision schedules to achieve advantageous efficiency and accuracy trade-offs, limiting their more comprehensive practical applications and achievable performance. To this end, we propose LDP, a Learnable Dynamic Precision DNN training framework that can automatically learn a temporally and spatially dynamic precision schedule during training towards optimal accuracy and efficiency trade-offs. It is worth noting that LDP-trained DNNs are by nature efficient during inference. Furthermore, we visualize the resulting temporal and spatial precision schedule and distribution of LDP trained DNNs on different tasks to better understand the corresponding DNNs' characteristics at different training stages and DNN layers both during and after training, drawing insights for promoting further innovations. Extensive experiments and ablation studies (seven networks, five datasets, and three tasks) show that the proposed LDP consistently outperforms state-of-the-art (SOTA) low precision DNN training techniques in terms of training efficiency and achieved accuracy trade-offs. For example, in addition to having the advantage of being automated, our LDP achieves a 0.31\% higher accuracy with a 39.1\% lower computational cost when training ResNet-20 on CIFAR-10 as compared with the best SOTA method.