S3: Increasing GPU Utilization during Generative Inference for Higher Throughput-猫眼课题宝

DOI：

10.48550/arxiv.2306.06000

发表时间：

2023-06

期刊：

ArXiv

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Yunho Jin;Chun-Feng Wu;D. Brooks;Gu-Yeon Wei

使用大型语言模型（LLM）生成文本会消耗大量内存。除了已经很大的模型参数之外，保存序列中先前令牌信息的键/值（KV）缓存可能会比模型本身更大。在当前的LLM服务框架中，由于不知道输出序列长度，该框架为KV缓存保留了最大序列长度的内存，以保证生成完整的序列。这限制了我们使用较小的批处理大小，从而降低GPU利用率，最重要的是降低吞吐量。我们认为设计一个具有输出序列先验知识的系统可以缓解这个问题。为此，我们提出了S$^{3}$，它预测输出序列长度，根据预测调度生成查询以提高设备资源利用率和吞吐量，并处理错误预测。我们提出的方法在假设输出序列长度为最坏情况的系统上实现了6.49$\times$吞吐量。

Generating texts with a large language model (LLM) consumes massive amounts of memory. Apart from the already-large model parameters, the key/value (KV) cache that holds information about previous tokens in a sequence can grow to be even larger than the model itself. This problem is exacerbated in one of the current LLM serving frameworks which reserves the maximum sequence length of memory for the KV cache to guarantee generating a complete sequence as they do not know the output sequence length. This restricts us to use a smaller batch size leading to lower GPU utilization and above all, lower throughput. We argue that designing a system with a priori knowledge of the output sequence can mitigate this problem. To this end, we propose S$^{3}$, which predicts the output sequence length, schedules generation queries based on the prediction to increase device resource utilization and throughput, and handle mispredictions. Our proposed method achieves 6.49$\times$ throughput over those systems that assume the worst case for the output sequence length.

S3: Increasing GPU Utilization during Generative Inference for Higher Throughput

文献摘要

相似文献