KRISP: Enabling Kernel-wise RIght-sizing for Spatial Partitioned GPU Inference Servers
KRISP: Enabling Kernel-wise RIght-sizing for Spatial Partitioned GPU Inference Servers
复制标题
DOI：
10.1109/hpca56546.2023.10071121
复制
发表时间：
2023-02
期刊：
影响因子：
--
通讯作者：
M. Chow;Ali Jahanshahi;Daniel Wong
中科院分区：
文献类型：
--
作者：
M. Chow;Ali Jahanshahi;Daniel Wong
机器学习（ML）推理工作负载与ML训练工作负载面临着显著不同的挑战。通常，推理工作负载运行时间较短，并且未充分利用GPU资源。为了克服这一点，已经提出了将模型的多个实例放在一起以提高GPU的利用率。协同定位的模型通过GPU空间分区设施共享GPU，例如Nvidia的MPS、GPU或AMD的CU Masking API。现有的空间分区的推理服务器创建模型明智的分区的基础上，一个模型的延迟容忍限制资源的“大小调整”。我们发现，模型明智的正确大小是未充分利用，由于不同的资源限制容忍的个别内核内的推理通过。我们提出了内核明智的正确大小的空间分区GPU推理服务器（KRISP），使内核明智的正确大小的空间分区在粒度的个别内核。我们证明，KRISP可以支持更高级别的并发运行的推理模型相比，现有的空间分区推理服务器。与孤立推理相比，KRISP将整体吞吐量提高了2倍（与之前的作品相比为1.22倍），并将每次推理的能耗降低了33%。
Machine learning (ML) inference workloads present significantly different challenges than ML training workloads. Typically, inference workloads are shorter running and under-utilize GPU resources. To overcome this, co-locating multiple instances of a model has been proposed to improve the utilization of GPUs. Co-located models share the GPU through GPU spatial partitioning facilities, such as Nvidia’s MPS, MIG, or AMD’s CU Masking API. Existing spatially partitioned inference servers create model-wise partitions by "right-sizing" based on a model’s latency tolerance to restricting resources. We show that model-wise right-sizing is under-utilized due to varying resource restriction tolerance of individual kernels within an inference pass.We propose Kernel-wise Right-sizing for Spatial Partitioned GPU Inference Servers (KRISP) to enable kernel-wise right-sizing of spatial partitions at the granularity of individual kernels. We demonstrate that KRISP can support a greater level of concurrently running inference models compared to existing spatially partitioned inference servers. KRISP improves overall throughput by 2x when compared with an isolated inference (1.22x vs prior works) and reduce energy per inference by 33%.