Hardware/Software Co-Exploration of Neural Architectures
Hardware/Software Co-Exploration of Neural Architectures
复制标题
DOI：
10.1109/tcad.2020.2986127
复制
发表时间：
2020-12-01
影响因子：
2.9
通讯作者：
Hu, Jingtong
中科院分区：
文献类型：
--
作者：
Jiang, Weiwen;Yang, Lei;Hu, Jingtong
我们提出了一个新的硬件和软件协同探索框架，用于高效的神经结构搜索（NAS）。与现有的硬件感知NAS假设一个固定的硬件设计并只探索NAS空间不同，我们的框架同时探索架构搜索空间和硬件设计空间，以确定最佳的神经网络架构和硬件对，从而最大化测试精度和硬件效率。这样的实践极大地开放了设计自由，并推动了硬件效率和测试精度之间的帕累托边界，以实现更好的设计权衡。框架迭代地执行两级（快速和慢速）探索。无需长时间的训练，快速探索可以有效地微调超参数，并在硬件规格方面修剪劣质架构，从而显著加快NAS进程。然后，缓慢探索在验证集上训练候选对象，并使用强化学习更新控制器，以最大限度地提高预期精度和硬件效率。在本文中，我们证明了协同探索框架可以有效地扩展搜索空间以包含高精度的模型，并且我们从理论上证明了所提出的两级优化可以有效地修剪劣质解以更好地探索搜索空间。在ImageNet上的实验结果表明，与硬件感知NAS相比，协同探索NAS能够以相同的准确率找到解决方案，吞吐量提高35.24%，能效提高54.05%。
We propose a novel hardware and software co-exploration framework for efficient neural architecture search (NAS). Different from existing hardware-aware NAS which assumes a fixed hardware design and explores the NAS space only, our framework simultaneously explores both the architecture search space and the hardware design space to identify the best neural architecture and hardware pairs that maximize both test accuracy and hardware efficiency. Such a practice greatly opens up the design freedom and pushes forward the Pareto frontier between hardware efficiency and test accuracy for better design tradeoffs. The framework iteratively performs a two-level (fast and slow) exploration. Without lengthy training, the fast exploration can effectively fine-tune hyperparameters and prune inferior architectures in terms of hardware specifications, which significantly accelerates the NAS process. Then, the slow exploration trains candidates on a validation set and updates a controller using the reinforcement learning to maximize the expected accuracy together with the hardware efficiency. In this article, we demonstrate that the co-exploration framework can effectively expand the search space to incorporate models with high accuracy, and we theoretically show that the proposed two-level optimization can efficiently prune inferior solutions to better explore the search space. The experimental results on ImageNet show that the co-exploration NAS can find solutions with the same accuracy, 35.24% higher throughput, 54.05% higher energy efficiency, compared with the hardware-aware NAS.