HALO: Hardware-Aware Learning to Optimize
HALO: Hardware-Aware Learning to Optimize
复制标题
DOI：
10.1007/978-3-030-58545-7_29
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
Chaojian Li;Tianlong Chen;Haoran You;Zhangyang Wang;Yingyan Lin
中科院分区：
文献类型：
--
作者：
Chaojian Li;Tianlong Chen;Haoran You;Zhangyang Wang;Yingyan Lin
将机器学习（ML）驱动的智能引入众多物联网（IoT）设备的需求呈爆炸式增长。然而，这种智能功能的有效性需要现场连续的模型适应以适应新的数据和环境，而设备上的计算和能源资源通常受到极大的限制。传统的手工制作（例如，SGD，Adagrad和Adam）和现有的Meta优化器都不是专门为应对这些挑战而设计的，因为前者需要繁琐的超参数调优，而后者由于Meta算法自身的开销而通常成本高昂。为此，我们提出了硬件感知学习优化（HALO），一个实用的Meta优化器，致力于资源高效的设备上的适应。我们的HALO优化器具有以下亮点：（1）更快的适应速度（即，采用较少的数据或迭代以达到指定的精度），通过引入新的正则化器来促进经验推广;以及（2）由于强制执行随机结构稀疏正则化器，降低了每次迭代的复杂度。此外，优化器本身被设计为一个非常轻量级的RNN，因此产生的开销可以忽略不计。在五个数据集、六个优化对象和两个最先进的（SOTA）边缘AI设备上进行的消融研究和实验证实，HALO在始终实现更好的准确性（0.46%-20.28%）的同时，可以大大降低适应的能源成本（高达60%），使用物联网设备或SOTA模拟器进行量化。代码和预训练模型位于 https://github.com/RICE-EIC/HALO .
There has been an explosive demand for bringing machine learning (ML) powered intelligence into numerous Internet-of-Things (IoT) devices. However, the effectiveness of such intelligent functionality requires in-situ continuous model adaptation for adapting to new data and environments, while the on-device computing and energy resources are usually extremely constrained. Neither traditional hand-crafted (e.g., SGD, Adagrad, and Adam) nor existing meta optimizers are specifically designed to meet those challenges, as the former requires tedious hyper-parameter tuning while the latter are often costly due to the meta algorithms’ own overhead. To this end, we proposehardware-aware learning to optimize(HALO), a practical meta optimizer dedicated to resource-efficient on-device adaptation. Our HALO optimizer features the following highlights: (1)faster adaptation speed(i.e., taking fewer data or iterations to reach a specified accuracy) by introducing a new regularizer to promote empirical generalization; and (2)lower per-iteration complexity, thanks to a stochastic structural sparsity regularizer being enforced. Furthermore, the optimizer itself is designed as a very light-weight RNN and thus incurs negligible overhead. Ablation studies and experiments on five datasets, six optimizees, and two state-of-the-art (SOTA) edge AI devices validate that, while always achieving a better accuracy (0.46% -20.28%), HALO can greatly trim down the energy cost (up to60%) in adaptation, quantified using an IoT device or SOTA simulator. Codes and pre-trained models are at https://github.com/RICE-EIC/HALO .