ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks-猫眼课题宝

DOI：

10.48550/arxiv.2205.08119

发表时间：

2022-05

期刊：

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Haoran You;Baopu Li;Huihong Shi;Y. Fu;Yingyan Lin

具有密集乘法的神经网络（NN）（例如，卷积和变压器）是有能力的，但耗电，阻碍了它们更广泛地部署到资源受限的设备中。因此，遵循节能硬件实现中的常见实践的无乘法网络使用更高效的算子（例如，逐位移位和加法）已经获得越来越多的关注。然而，无乘法网络在实现的准确性方面通常表现不佳。为此，这项工作提倡混合神经网络，包括强大但昂贵的乘法和有效但功能较弱的运营商，以结合两个世界的最佳，并提出ShiftAddNAS，它可以自动搜索更准确，更有效的神经网络。我们的ShiftAddNAS突出了两个推动因素。具体而言，它集成了（1）第一混合搜索空间，该混合搜索空间包含基于乘法和无乘法的算子，以促进准确和有效的混合NN的开发;以及（2）一种新的权重共享策略，该策略能够在遵循异构分布的不同算子之间进行有效的权重共享（例如，高斯卷积与拉普拉斯加法运算），同时导致超网规模大大减少，搜索网络更好。对各种模型，数据集和任务的广泛实验和消融研究一致验证了ShiftAddNAS的功效，例如，与最先进的NN相比，准确度提高了+7.7%，BLEU得分提高了+4.9，同时分别节省了高达93%或69%的能量和延迟。代码和预训练模型可在https://github.com/RICE-EIC/ShiftAddNAS上获得。

Neural networks (NNs) with intensive multiplications (e.g., convolutions and transformers) are capable yet power hungry, impeding their more extensive deployment into resource-constrained devices. As such, multiplication-free networks, which follow a common practice in energy-efficient hardware implementation to parameterize NNs with more efficient operators (e.g., bitwise shifts and additions), have gained growing attention. However, multiplication-free networks usually under-perform their vanilla counterparts in terms of the achieved accuracy. To this end, this work advocates hybrid NNs that consist of both powerful yet costly multiplications and efficient yet less powerful operators for marrying the best of both worlds, and proposes ShiftAddNAS, which can automatically search for more accurate and more efficient NNs. Our ShiftAddNAS highlights two enablers. Specifically, it integrates (1) the first hybrid search space that incorporates both multiplication-based and multiplication-free operators for facilitating the development of both accurate and efficient hybrid NNs; and (2) a novel weight sharing strategy that enables effective weight sharing among different operators that follow heterogeneous distributions (e.g., Gaussian for convolutions vs. Laplacian for add operators) and simultaneously leads to a largely reduced supernet size and much better searched networks. Extensive experiments and ablation studies on various models, datasets, and tasks consistently validate the efficacy of ShiftAddNAS, e.g., achieving up to a +7.7% higher accuracy or a +4.9 better BLEU score compared to state-of-the-art NN, while leading to up to 93% or 69% energy and latency savings, respectively. Codes and pretrained models are available at https://github.com/RICE-EIC/ShiftAddNAS.

ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks

文献摘要

相似文献