CSR: SMALL: Low-Latency Model Inference Using Cellular Batching

CSR：SMALL：使用蜂窝批处理的低延迟模型推理

• 批准号: 1816717
• 负责人: Jinyang Li
• 金额: $ 41.13万
• 依托单位: New York University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-09-01 至 2021-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1816717&HistoricalAwards=false
• 关键词: CSR; SMALL; Low; Latency; Model

Successful cloud deployment of machine learning services, such as language translation, image search and home assistants require a high performance serving system that can process hundreds of thousands requests per second. It is particularly crucial for the serving system to ensure low latency, as even tens of milliseconds increase in delays can annoy users when using a service like the home assistant. Among the widely-used deep learning models, recurrent neural network (RNN) is an important class of models that incur high latency when processed by existing serving systems. This project aims to develop a new serving system that can handle a variety of Artificial Intelligence (AI) tasks using RNN-based deep learning models with significantly improved latency.To achieve good throughput on modern hardware, one must perform batched computation. This project develops a new, dynamic approach to batching, called Cellular Batching. Cellular Batching performs batching and execution at the granularity of a "cell" (aka a subgraph with embedded model weights) instead of the entire dataflow graph, as is done in existing systems. Under Cellular Batching, a new request can immediately join the execution of ongoing requests to minimize queuing delays and increase effective batching. The project will complete research tasks that make Cellular Batching practical (by developing an efficient scheduler and supporting zero-downtime model upgrading) and generalize it to different models such as search-guided RNNs.Deep learning models based on RNNs are becoming widely used to accomplish various AI tasks ranging from speech recognition and language translation, to question answering. As such, there is a pressing demand for a high-throughput and low-latency serving system, in order to improve end-user experience and reduce the cost of deployment. By demonstrating significant latency and throughput benefits, there is high potential for Cellular Batching to be widely adopted. This project will also develop a new course component on high performance machine learning systems as part of the graduate-level distributed systems course.This project will produce data in the form of source code, various serving benchmarks, and experimental results. The source code and all benchmarks used in the experiments will be distributed via Github. A local copy of the source code and the publications produced by the project will also be made available at the URL (http://batchmaker.news.cs.nyu.edu) for at least three years beyond the award period.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

成功的云部署机器学习服务，例如语言翻译，图像搜索和家庭助理需要高性能服务系统，该系统每秒可以处理数十万个请求。 对于服务系统来说，确保低潜伏期尤为重要，因为在使用家庭助理等服务时，即使延迟数十毫秒的延迟也可能会使用户烦恼。 在广泛使用的深度学习模型中，经常性的神经网络（RNN）是一类重要的模型，在现有服务系统处理时会产生高潜伏期。 该项目旨在开发一个新的服务系统，该系统可以使用基于RNN的深度学习模型来处理各种人工智能（AI）任务，具有显着改善的延迟。要实现现代硬件的良好吞吐量，必须执行批处理计算。 该项目开发了一种新的动态批处理方法，称为蜂窝批处理。 蜂窝批处理在“单元格”（又称具有嵌入式模型权重的子图）的粒度上执行批处理和执行，而不是整个数据流图，就像在现有系统中所做的那样。 在蜂窝批处理下，新请求可以立即加入执行正在进行的请求，以最大程度地减少排队延迟并增加有效的批处理。 该项目将完成使细胞批处理实用的研究任务（通过开发有效的调度程序并支持零downtime模型升级），并将其推广到不同模型，例如搜索引导的RNNS.Deep学习模型，基于RNNS的深度学习模型正被广泛用于完成从语音识别和语言转换的各种AI任务，以提出反应。 因此，为了提高最终用户体验并降低部署成本，对高通量和低延迟服务系统有紧迫的需求。 通过证明明显的延迟和吞吐量益处，可以广泛采用细胞批处理的潜力。 作为研究生级分布式系统课程的一部分，该项目还将在高性能机器学习系统上开发一个新的课程组件。该项目将以源代码，各种服务基准和实验结果的形式生成数据。 实验中使用的源代码和所有基准将通过GitHub分发。 A local copy of the source code and the publications produced by the project will also be made available at the URL (http://batchmaker.news.cs.nyu.edu) for at least three years beyond the award period.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

项目成果

Supporting Very Large Models using Automatic Dataflow Graph Partitioning

• DOI: 10.1145/3302424.3303953
• 发表时间: 2018-07
• 期刊: Proceedings of the Fourteenth EuroSys Conference 2019
• 作者: Minjie Wang;Chien-chin Huang;Jinyang Li

SwapAdvisor: Pushing Deep Learning Beyond the GPU Memory Limit via Smart Swapping

• DOI: 10.1145/3373376.3378530
• 发表时间: 2020-03
• 期刊: Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems
• 作者: Chien-chin Huang;Gu Jin;Jinyang Li

Low latency RNN inference with cellular batching

• DOI: 10.1145/3190508.3190541
• 发表时间: 2018-04
• 期刊: Proceedings of the Thirteenth EuroSys Conference
• 作者: Pin Gao;Lingfan Yu;Yongwei Wu;Jinyang Li

Deep Graph Library: Towards Efficient and Scalable Deep Learning on Graphs

• 发表时间: 2019-09
• 期刊: ArXiv
• 作者: Minjie Wang;Lingfan Yu;Da Zheng;Quan Gan;Yujie Gai;Zihao Ye;Mufei Li;Jinjing Zhou;Qi Huang-