CSR: SMALL: Low-Latency Model Inference Using Cellular Batching

CSR：SMALL：使用蜂窝批处理的低延迟模型推理

• 批准号: 1816717
• 负责人: Jinyang Li
• 金额: $ 41.13万
• 依托单位: New York University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-09-01 至 2021-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1816717&HistoricalAwards=false
• 关键词: CSR; SMALL; Low; Latency; Model

Successful cloud deployment of machine learning services, such as language translation, image search and home assistants require a high performance serving system that can process hundreds of thousands requests per second. It is particularly crucial for the serving system to ensure low latency, as even tens of milliseconds increase in delays can annoy users when using a service like the home assistant. Among the widely-used deep learning models, recurrent neural network (RNN) is an important class of models that incur high latency when processed by existing serving systems. This project aims to develop a new serving system that can handle a variety of Artificial Intelligence (AI) tasks using RNN-based deep learning models with significantly improved latency.To achieve good throughput on modern hardware, one must perform batched computation. This project develops a new, dynamic approach to batching, called Cellular Batching. Cellular Batching performs batching and execution at the granularity of a "cell" (aka a subgraph with embedded model weights) instead of the entire dataflow graph, as is done in existing systems. Under Cellular Batching, a new request can immediately join the execution of ongoing requests to minimize queuing delays and increase effective batching. The project will complete research tasks that make Cellular Batching practical (by developing an efficient scheduler and supporting zero-downtime model upgrading) and generalize it to different models such as search-guided RNNs.Deep learning models based on RNNs are becoming widely used to accomplish various AI tasks ranging from speech recognition and language translation, to question answering. As such, there is a pressing demand for a high-throughput and low-latency serving system, in order to improve end-user experience and reduce the cost of deployment. By demonstrating significant latency and throughput benefits, there is high potential for Cellular Batching to be widely adopted. This project will also develop a new course component on high performance machine learning systems as part of the graduate-level distributed systems course.This project will produce data in the form of source code, various serving benchmarks, and experimental results. The source code and all benchmarks used in the experiments will be distributed via Github. A local copy of the source code and the publications produced by the project will also be made available at the URL (http://batchmaker.news.cs.nyu.edu) for at least three years beyond the award period.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

机器学习服务的成功云部署，如语言翻译、图像搜索和家庭助理，需要一个每秒可以处理数十万个请求的高性能服务系统。 对于服务系统来说，确保低延迟尤其重要，因为即使延迟增加数十毫秒，在使用家庭助理等服务时也会惹恼用户。 在广泛使用的深度学习模型中，递归神经网络（RNN）是一类重要的模型，在由现有服务系统处理时会产生高延迟。 该项目旨在开发一种新的服务系统，该系统可以使用基于RNN的深度学习模型处理各种人工智能（AI）任务，并显着改善延迟。为了在现代硬件上实现良好的吞吐量，必须执行批处理计算。 该项目开发了一种新的动态方法来进行批量处理，称为Cellular Batching。 Cellular Batching以“cell”（也称为具有嵌入式模型权重的子图）的粒度执行优化和执行，而不是像现有系统中所做的那样，以整个优化图执行。 在Cellular Batching下，新的请求可以立即加入正在进行的请求的执行，以最大限度地减少排队延迟并增加有效的排队时间。 该项目将完成使Cellular Batching实用化的研究任务（通过开发高效的调度器和支持零停机模型升级），并将其推广到不同的模型，如搜索引导的RNN。基于RNN的深度学习模型正被广泛用于完成从语音识别和语言翻译到问答的各种人工智能任务。 因此，迫切需要高吞吐量和低延迟的服务系统，以改善最终用户体验并降低部署成本。 通过展示显着的延迟和吞吐量优势，Cellular Batching有很大的潜力被广泛采用。 该项目还将开发一个新的高性能机器学习系统课程组件，作为研究生级分布式系统课程的一部分。该项目将以源代码、各种服务基准和实验结果的形式生成数据。 实验中使用的源代码和所有基准测试将通过Github分发。 该项目的源代码和出版物的本地副本也将在URL（http：//www.example.com）上提供，至少在授予期后的三年内。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。batchmaker.news.cs.nyu.edu

项目成果

Supporting Very Large Models using Automatic Dataflow Graph Partitioning

• DOI: 10.1145/3302424.3303953
• 发表时间: 2018-07
• 期刊: Proceedings of the Fourteenth EuroSys Conference 2019
• 作者: Minjie Wang;Chien-chin Huang;Jinyang Li

SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping

• DOI: 10.1145/3373376.3378530
• 发表时间: 2020-03
• 期刊: Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems
• 作者: Chien-chin Huang;Gu Jin;Jinyang Li

Deep Graph Library: Towards Efficient and Scalable Deep Learning on Graphs

• 发表时间: 2019-09
• 期刊: ArXiv
• 作者: Minjie Wang;Lingfan Yu;Da Zheng;Quan Gan;Yujie Gai;Zihao Ye;Mufei Li;Jinjing Zhou;Qi Huang-

Low latency RNN inference with cellular batching

• DOI: 10.1145/3190508.3190541
• 发表时间: 2018-04
• 期刊: Proceedings of the Thirteenth EuroSys Conference
• 作者: Pin Gao;Lingfan Yu;Yongwei Wu;Jinyang Li