E2-Train: Training State-of-the-art CNNs with Over 80% Energy Savings
E2-Train: Training State-of-the-art CNNs with Over 80% Energy Savings
复制标题
DOI：
--
复制
发表时间：
2019-10
期刊：
影响因子：
--
通讯作者：
Yue Wang;Ziyu Jiang;Xiaohan Chen;Pengfei Xu;Yang Zhao;Yingyan Lin;Zhangyang Wang
中科院分区：
文献类型：
--
作者：
Yue Wang;Ziyu Jiang;Xiaohan Chen;Pengfei Xu;Yang Zhao;Yingyan Lin;Zhangyang Wang
卷积神经网络 (CNN) 已越来越多地部署到边缘设备。因此，为了在资源受限的平台上进行高效的 CNN 推理，人们做出了许多努力。本文试图探索一个正交方向：如何对CNN进行更节能的训练，从而实现设备上训练。我们努力通过从三个互补级别删除不必要的计算来降低训练期间的能源成本：数据级别的随机小批量删除；模型级别的选择性层更新；以及算法层面的低成本、低精度反向传播的符号预测。广泛的模拟和消融研究以及 FPGA 板的实际能量测量，证实了我们提出的策略的优越性，并证明了训练的显着节能效果。例如，当在 CIFAR-10 上训练 ResNet-74 时，我们实现了 >90% 和 >60% 的大幅节能，而 top-1 准确度损失分别仅为约 2% 和 1.2%。在 CIFAR-100 上训练 ResNet-110 时，在不降低推理精度的情况下实现了超过 84% 的训练节能。
Convolutional neural networks (CNNs) have been increasingly deployed to edge devices. Hence, many efforts have been made towards efficient CNN inference in resource-constrained platforms. This paper attempts to explore an orthogonal direction: how to conduct more energy-efficient training of CNNs, so as to enable on-device training. We strive to reduce the energy cost during training, by dropping unnecessary computations from three complementary levels: stochastic mini-batch dropping on the data level; selective layer update on the model level; and sign prediction for low-cost, low-precision back-propagation, on the algorithm level. Extensive simulations and ablation studies, with real energy measurements from an FPGA board, confirm the superiority of our proposed strategies and demonstrate remarkable energy savings for training. For example, when training ResNet-74 on CIFAR-10, we achieve aggressive energy savings of >90% and >60%, while incurring a top-1 accuracy loss of only about 2% and 1.2%, respectively. When training ResNet-110 on CIFAR-100, an over 84% training energy saving is achieved without degrading inference accuracy.