Bolt: Bridging the Gap between Auto-tuners and Hardware-native Performance
Bolt: Bridging the Gap between Auto-tuners and Hardware-native Performance
复制标题
DOI：
--
复制
发表时间：
2021-10
期刊：
影响因子：
--
通讯作者：
Jiarong Xing;Leyuan Wang-;Shang Zhang;Jack H Chen;Ang Chen;Yibo Zhu
中科院分区：
文献类型：
--
作者：
Jiarong Xing;Leyuan Wang-;Shang Zhang;Jack H Chen;Ang Chen;Yibo Zhu
今天的自动调谐器（例如，AutoTVM，Ansor）通过导航一个大的搜索空间来识别有效的实现，从而生成有效的张量程序，但它们这样做的硬件细节是不透明的。因此，它们的性能可能落后于硬件本地库（例如，cuBLAS、cuDNN），它们由设备供应商手动优化以获得高性能。另一方面，这些供应商库有一组固定的支持功能，缺乏自动调谐器提供的定制和自动化支持。Bolt是基于最近的趋势，即供应商库越来越模块化，并通过声明式控制（例如，CUTLASS）。它启用了一种新颖的方法，通过硬件本地模板搜索来弥合这一差距并实现两全其美。Bolt提供了新的机会，可以在图、运算符和模型级别重新思考端到端张量优化。Bolt通过在TVM中流行的自动调谐器和一类广泛使用的平台（即，NVIDIA GPU）--两者都在我们的生产环境中进行了大规模部署。Bolt将普通卷积神经网络的推理速度平均提高了2.5倍，并在20分钟内自动调整这些模型。
Today's auto-tuners (e.g., AutoTVM, Ansor) generate efficient tensor programs by navigating a large search space to identify effective implementations, but they do so with opaque hardware details. Thus, their performance could fall behind that of hardware-native libraries (e.g., cuBLAS, cuDNN), which are hand-optimized by device vendors to extract high performance. On the other hand, these vendor libraries have a fixed set of supported functions and lack the customization and automation support afforded by auto-tuners. Bolt is based on the recent trend that vendor libraries are increasingly modularized and reconfigurable via declarative control (e.g., CUTLASS). It enables a novel approach that bridges this gap and achieves the best of both worlds, via hardware-native templated search. Bolt provides new opportunities to rethink end-to-end tensor optimizations at the graph, operator, and model levels. Bolt demonstrates this concept by prototyping on a popular auto-tuner in TVM and a class of widely-used platforms (i.e., NVIDIA GPUs) -- both in large deployment in our production environment. Bolt improves the inference speed of common convolutional neural networks by 2.5x on average over the state of the art, and it auto-tunes these models within 20 minutes.