HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark
HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark
复制标题
DOI：
--
复制
发表时间：
2021-03
期刊：
影响因子：
--
通讯作者：
Chaojian Li;Zhongzhi Yu;Yonggan Fu;Yongan Zhang;Yang Zhao;Haoran You;Qixuan Yu;Yue Wang;Yingyan Lin
中科院分区：
文献类型：
--
作者：
Chaojian Li;Zhongzhi Yu;Yonggan Fu;Yongan Zhang;Yang Zhao;Haoran You;Qixuan Yu;Yue Wang;Yingyan Lin
硬件感知神经架构搜索（HW - NAS）最近通过自动化在更多资源受限的日常生活设备中部署的深度神经网络（DNN）设计而受到了极大关注。尽管其性能前景可观，但开发最优的HW - NAS解决方案可能极具挑战性，因为它需要算法、微架构和特定设备编译方面的跨学科知识。首先，为了确定要纳入NAS过程的硬件成本，现有工作大多采用预先收集的硬件成本查找表或特定设备的硬件成本模型。这两者都限制了HW - NAS创新的发展，并对非硬件专家构成了进入壁垒。其次，与通用NAS类似，由于HW - NAS算法所需的大量计算资源以及所采用的搜索空间、超参数和硬件设备的差异，对其进行基准测试可能极其困难。为此，我们开发了HW - NAS - Bench，这是第一个用于HW - NAS研究的公共数据集，旨在使非硬件专家也能进行HW - NAS研究，并使HW - NAS研究更具可重复性和可获取性。为了设计HW - NAS - Bench，我们仔细收集了NAS - Bench - 201和FBNet搜索空间中所有网络在六种硬件设备上的实测/估算硬件性能，这些设备分为三类（即商业边缘设备、FPGA和ASIC）。此外，我们对HW - NAS - Bench中收集的测量数据进行了全面分析，为HW - NAS研究提供了见解。最后，我们展示了示例用户案例，以（1）表明HW - NAS - Bench允许非硬件专家通过简单查询来进行HW - NAS，以及（2）验证特定于设备的专用HW - NAS确实能够实现最优的精度 - 成本权衡。代码和所有收集的数据可在https://github.com/RICE - EIC/HW - NAS - Bench获取。
HardWare-aware Neural Architecture Search (HW-NAS) has recently gained tremendous attention by automating the design of DNNs deployed in more resource-constrained daily life devices. Despite its promising performance, developing optimal HW-NAS solutions can be prohibitively challenging as it requires cross-disciplinary knowledge in the algorithm, micro-architecture, and device-specific compilation. First, to determine the hardware-cost to be incorporated into the NAS process, existing works mostly adopt either pre-collected hardware-cost look-up tables or device-specific hardware-cost models. Both of them limit the development of HW-NAS innovations and impose a barrier-to-entry to non-hardware experts. Second, similar to generic NAS, it can be notoriously difficult to benchmark HW-NAS algorithms due to their significant required computational resources and the differences in adopted search spaces, hyperparameters, and hardware devices. To this end, we develop HW-NAS-Bench, the first public dataset for HW-NAS research which aims to democratize HW-NAS research to non-hardware experts and make HW-NAS research more reproducible and accessible. To design HW-NAS-Bench, we carefully collected the measured/estimated hardware performance of all the networks in the search spaces of both NAS-Bench-201 and FBNet, on six hardware devices that fall into three categories (i.e., commercial edge devices, FPGA, and ASIC). Furthermore, we provide a comprehensive analysis of the collected measurements in HW-NAS-Bench to provide insights for HW-NAS research. Finally, we demonstrate exemplary user cases to (1) show that HW-NAS-Bench allows non-hardware experts to perform HW-NAS by simply querying it and (2) verify that dedicated device-specific HW-NAS can indeed lead to optimal accuracy-cost trade-offs. The codes and all collected data are available at https://github.com/RICE-EIC/HW-NAS-Bench.