CPT: Efficient Deep Neural Network Training via Cyclic Precision-猫眼课题宝

DOI：

发表时间：

2021-01

期刊：

ArXiv

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Y. Fu;Han Guo;Meng Li;Xin Yang;Yining Ding;V. Chandra;Yingyan Lin

低精度深度神经网络（DNN）训练已引起极大的关注，因为降低精度是提高DNNS训练时间/能源效率的最有效旋钮之一。在本文中，我们试图从新的角度探索低精度培训，这是受到了解DNN培训的最新发现的启发：我们认为，DNNS的精度可能与DNN培训期间的学习率相似，并提倡沿培训轨迹的动态精度，以进一步提高DNN培训的时间/能源效率。具体而言，我们提出循环精确训练（CPT），以周期性地改变两个边界值之间的精度，可以在前几个训练时期使用简单的精确范围测试来识别，这些边界值可以识别。对五个数据集和11个模型的大量模拟和消融研究表明，CPT在各种模型/任务（包括分类和语言建模）中的有效性是一致的。此外，通过实验和可视化，我们表明CPT有助于（1）收敛到更较大的最小值，并具有较低的概括误差，并且（2）减少训练方差，我们相信我们认为，这为新的设计旋钮提供了同时提高DNN训练的优化和效率的新设计旋钮。我们的代码可用：此HTTPS URL

Low-precision deep neural network (DNN) training has gained tremendous attention as reducing precision is one of the most effective knobs for boosting DNNs' training time/energy efficiency. In this paper, we attempt to explore low-precision training from a new perspective as inspired by recent findings in understanding DNN training: we conjecture that DNNs' precision might have a similar effect as the learning rate during DNN training, and advocate dynamic precision along the training trajectory for further boosting the time/energy efficiency of DNN training. Specifically, we propose Cyclic Precision Training (CPT) to cyclically vary the precision between two boundary values which can be identified using a simple precision range test within the first few training epochs. Extensive simulations and ablation studies on five datasets and eleven models demonstrate that CPT's effectiveness is consistent across various models/tasks (including classification and language modeling). Furthermore, through experiments and visualization we show that CPT helps to (1) converge to a wider minima with a lower generalization error and (2) reduce training variance which we believe opens up a new design knob for simultaneously improving the optimization and efficiency of DNN training. Our codes are available at: this https URL

CPT: Efficient Deep Neural Network Training via Cyclic Precision

文献摘要

相似文献