Overflowing emerging neural network inference tasks from the GPU to the CPU on heterogeneous servers
Overflowing emerging neural network inference tasks from the GPU to the CPU on heterogeneous servers
复制标题
DOI：
10.1145/3534056.3534935
复制
发表时间：
2022-06
期刊：
影响因子：
--
通讯作者：
Adithya Kumar;A. Sivasubramaniam;T. Zhu
中科院分区：
文献类型：
--
作者：
Adithya Kumar;A. Sivasubramaniam;T. Zhu
虽然当前的深度学习（DL）推理运行时系统根据其能力将模型的任务依次卸载到可用的GPU/加速器上，但我们提出了一个选择性地将其中一些任务重定向到CPU并与GPU并行运行它们的案例。这种新的机会特别出现在新兴的DL模型中，其数据流图（DFG）与传统的任务线性链相比具有更宽的扇出。通过将这些任务中的一些转移到CPU，我们可以（i）从DFG的关键路径中减少服务时间，（ii）将GPU用于更值得的任务，以及（iii）提高服务器中配置硬件的整体利用率。然而，诸如其在DFG中的关键性、移动到不同硬件引擎时的减速以及跨这些引擎传输输入/输出数据的开销等几个因素决定了要定向什么/何时/如何定向任务。虽然这对计算要求很高，并且解决起来很慢，但通过一系列的原理，我们得出了一种快速的技术，用于从GPU到CPU的任务溢出。我们在一个敏捷的异构并发运行时引擎上实现了这项技术，该引擎构建在最先进的ONNXtron引擎之上，并展示了延迟减少> 10%，吞吐量增加> 19%，以及新兴神经网络模型的GPU内存使用节省> 9.8%。
While current deep learning (DL) inference runtime systems sequentially offload the model's tasks on to an available GPU/accelerator based on its capability, we make a case for selectively redirecting some of these tasks to the CPU and running them concurrently with the GPU doing other work. This new opportunity specifically arises for emerging DL models whose data flow graphs (DFGs) have much wider fan-outs compared to traditional ones which are invariably linear chains of tasks. By opportunistically moving some of these tasks to the CPU, we can (i) shave off service times from the critical path of the DFG, (ii) devote the GPU for more deserving tasks, and (iii) improve overall utilization of the provisioned hardware in the server. However, several factors such as its criticality in the DFG, slowdown when moved to a different hardware engine, and overheads in transferring input/output data across these engines, determine the what/when/how of tasks to be directed. While this is computationally demanding and slow to be solved optimally, through a series of rationales we derive a fast technique for task overflow from GPU to CPU. We implement this technique on a nimble heterogeneous concurrent runtime engine built on top of the state-of-the-art ONNXRuntime engine and demonstrate > 10% reduction in latency, > 19% gain in throughput, and > 9.8% savings in GPU memory usage for emerging neural network models.