CSR: SMALL: Low-Latency Model Inference Using Cellular Batching

成功的云部署机器学习服务，如语言翻译、图像搜索和家庭助理，需要一个高性能的服务系统，每秒可以处理数十万个请求。对于服务系统来说，确保低延迟尤为重要，因为即使延迟增加几十毫秒也会让用户在使用家庭助理等服务时感到烦恼。在广泛使用的深度学习模型中，递归神经网络（RNN）是一类重要的模型，在现有的服务系统处理时会产生很高的延迟。该项目旨在开发一种新的服务系统，该系统可以使用基于rnn的深度学习模型处理各种人工智能（AI）任务，并显著改善延迟。为了在现代硬件上实现良好的吞吐量，必须执行批处理计算。本项目开发了一种新的、动态的批处理方法，称为蜂窝式批处理。蜂窝批处理以“单元”（即具有嵌入式模型权重的子图）的粒度执行批处理和执行，而不是像在现有系统中那样对整个数据流图执行批处理。在蜂窝式批处理中，新请求可以立即加入正在执行的请求的执行，以最小化排队延迟并增加有效的批处理。该项目将完成使蜂窝批处理实用的研究任务（通过开发有效的调度程序和支持零停机模型升级），并将其推广到不同的模型，如搜索引导rnn。基于rnn的深度学习模型正被广泛用于完成各种人工智能任务，从语音识别、语言翻译到问题回答。因此，迫切需要高吞吐量和低延迟的服务系统，以改善最终用户体验并降低部署成本。通过展示显著的延迟和吞吐量优势，蜂窝批处理被广泛采用的潜力很大。该项目还将开发一个关于高性能机器学习系统的新课程组件，作为研究生级分布式系统课程的一部分。该项目将以源代码、各种服务基准和实验结果的形式生成数据。实验中使用的源代码和所有基准测试将通过Github发布。该项目的源代码和出版物的本地副本也将在网址（http://batchmaker.news.cs.nyu.edu）上提供，期限至少为三年。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Successful cloud deployment of machine learning services, such as language translation, image search and home assistants require a high performance serving system that can process hundreds of thousands requests per second. It is particularly crucial for the serving system to ensure low latency, as even tens of milliseconds increase in delays can annoy users when using a service like the home assistant. Among the widely-used deep learning models, recurrent neural network (RNN) is an important class of models that incur high latency when processed by existing serving systems. This project aims to develop a new serving system that can handle a variety of Artificial Intelligence (AI) tasks using RNN-based deep learning models with significantly improved latency.To achieve good throughput on modern hardware, one must perform batched computation. This project develops a new, dynamic approach to batching, called Cellular Batching. Cellular Batching performs batching and execution at the granularity of a "cell" (aka a subgraph with embedded model weights) instead of the entire dataflow graph, as is done in existing systems. Under Cellular Batching, a new request can immediately join the execution of ongoing requests to minimize queuing delays and increase effective batching. The project will complete research tasks that make Cellular Batching practical (by developing an efficient scheduler and supporting zero-downtime model upgrading) and generalize it to different models such as search-guided RNNs.Deep learning models based on RNNs are becoming widely used to accomplish various AI tasks ranging from speech recognition and language translation, to question answering. As such, there is a pressing demand for a high-throughput and low-latency serving system, in order to improve end-user experience and reduce the cost of deployment. By demonstrating significant latency and throughput benefits, there is high potential for Cellular Batching to be widely adopted. This project will also develop a new course component on high performance machine learning systems as part of the graduate-level distributed systems course.This project will produce data in the form of source code, various serving benchmarks, and experimental results. The source code and all benchmarks used in the experiments will be distributed via Github. A local copy of the source code and the publications produced by the project will also be made available at the URL (http://batchmaker.news.cs.nyu.edu) for at least three years beyond the award period.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.