DOPpler: Parallel Measurement Infrastructure for Auto-Tuning Deep Learning Tensor Programs
DOPpler: Parallel Measurement Infrastructure for Auto-Tuning Deep Learning Tensor Programs
复制标题
DOI：
10.1109/tpds.2023.3279233
复制
发表时间：
2023-07
影响因子：
5.3
通讯作者：
Damian Borowiec;Gingfung Yeung;A. Friday;Richard Harper;Peter Garraghan
中科院分区：
文献类型：
--
作者：
Damian Borowiec;Gingfung Yeung;A. Friday;Richard Harper;Peter Garraghan
深度学习模型、库和硬件的异构性对提高模型推理性能构成了重要挑战。自动调谐器通过针对目标设备的自动张量程序优化来解决这一挑战。然而，自动调谐器的完成会产生相当大的时间成本，因为它们的设计需要在隔离的目标设备内连续执行张量程序候选测量，以将延迟测量误差降至最低。在本文中，我们提出了一种并行自动调谐测量基础设施--多普勒。与传统方法相比，多普勒允许相当大的自动调整加速比，同时保持高质量的张量程序优化。通过提出并行执行引擎来高效地在CPU主机和GPU目标设备上并行执行候选张量程序，并通过在测量张量程序内核延迟时引入高精度的在设备测量技术来克服测量不准确，从而加速了自动调优过程。多普勒旨在自动计算最佳并行度，以便为不同的张量程序、自动调谐器和目标设备提供快速准确的自动调谐。实验结果表明，在获得相当于传统自动调谐基础设施的优化增益的同时，多普勒平均使总自动调谐时间减少了50.5%。
The heterogeneity of Deep Learning models, libraries, and hardware poses an important challenge for improving model inference performance. Auto-tuners address this challenge via automatic tensor program optimization towards a target-device. However, auto-tuners incur a substantial time cost to complete given their design necessitates performing tensor program candidate measurements serially within an isolated target-device to minimize latency measurement inaccuracy. In this article we propose DOPpler, a parallel auto-tuning measurement infrastructure. DOPpler allows for considerable auto-tuning speedup over conventional approaches whilst maintaining high-quality tensor program optimization. DOPpler accelerates the auto-tuning process by proposing a parallel execution engine to efficiently execute candidate tensor programs in parallel across the CPU-host and GPU target-device, and overcomes measurement inaccuracy by introducing a high-precision on-device measurement technique when measuring tensor program kernel latency. DOPpler is designed to automatically calculate the optimal degree of parallelism to provision fast and accurate auto-tuning for different tensor programs, auto-tuners and target-devices. Experiment results show that DOPpler reduces total auto-tuning time by 50.5% on average whilst achieving optimization gains equivalent to conventional auto-tuning infrastructure.