Low latency RNN inference with cellular batching
Low latency RNN inference with cellular batching
复制标题
DOI：
10.1145/3190508.3190541
复制
发表时间：
2018-04
期刊：
影响因子：
--
通讯作者：
Pin Gao;Lingfan Yu;Yongwei Wu;Jinyang Li
中科院分区：
文献类型：
--
作者：
Pin Gao;Lingfan Yu;Yongwei Wu;Jinyang Li
在预训练的神经网络模型上执行推理必须满足低延迟的要求，这通常与实现高吞吐量不一致。现有的深度学习系统使用递归神经网络来提高吞吐量，这在为具有动态递归图的递归神经网络提供服务时表现不佳。我们提出了蜂窝网络的技术，它提高了延迟和吞吐量的RNN推理。与现有的批量处理一组固定的递归图的系统不同，蜂窝递归在RNN“单元”（具有共享权重的子图）的粒度上做出递归决策，并在请求加入和离开系统时动态组装一个批量单元以供执行。我们在一个名为BatchMaker的系统中实现了我们的方法。实验表明，BatchMaker实现了更低的延迟和更高的吞吐量比现有的系统。
Performing inference on pre-trained neural network models must meet the requirement of low-latency, which is often at odds with achieving high throughput. Existing deep learning systems use batching to improve throughput, which do not perform well when serving Recurrent Neural Networks with dynamic dataflow graphs. We propose the technique of cellular batching, which improves both the latency and throughput of RNN inference. Unlike existing systems that batch a fixed set of dataflow graphs, cellular batching makes batching decisions at the granularity of an RNN "cell" (a subgraph with shared weights) and dynamically assembles a batched cell for execution as requests join and leave the system. We implemented our approach in a system called BatchMaker. Experiments show that BatchMaker achieves much lower latency and also higher throughput than existing systems.