CRII: CSR: Mitigating Tail Latency in Prediction-serving Systems using Coding-theoretic Tools

预测服务系统接受来自机器学习模型的查询和返回预测。例如，图像分类服务将把一张图像作为查询，并以预测图像中是否有汽车作为响应。预测服务系统在当今的各种应用中变得越来越重要。这些系统通常在大型计算基础设施中运行，在那里故障和速度减慢是常见的。这种不可用导致查询-响应延迟显著增加，从而降低了服务质量。该研究项目设计并实现了一种资源高效的预测服务系统健壮性解决方案，该解决方案使用了编码理论领域中的一种工具--擦除编码计算。该项目的总体目标是设计并实现一种解决方案，以减少使用擦除编码计算的预测服务系统中潜在的长延迟。该项目通过以一种新颖的方式使用机器学习，克服了现有编码计算解决方案中的关键限制。具体来说，该项目涉及以下关键任务：(1)设计一种低开销的编码计算解决方案，通过采用基于学习的方法有效地支持复杂的非线性函数；(2)设计一种符合真实世界预测服务系统约束的高效训练方法；以及(3)系统设计和实现。该项目产生的软件将被集成到开源预测服务系统中，用于研究和教育。将继续积极参与促进科学、技术、工程和数学学科的多样性，并提供本科生指导。该项目的发现将被整合到计算机科学的研究生和本科课程中。研究项目的结果，包括软件，将在http://www.cs.cmu.edu/~rvinayak/crii上提供。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Prediction-serving systems take in queries and return predictions from a machine learning model. For example, an image classification service would take an image as a query and respond with a prediction of whether there is a car in the image. Prediction-serving systems are increasingly important for a wide variety of applications today. These systems are typically run in large-scale computing infrastructures, where failures and slowdowns are common. Such unavailability results in significant increase in the query-response latency, thereby degrading the quality-of-service. The research project designs and implements a resource-efficient solution for robustness in prediction-serving systems using a tool from the domain of coding theory called "erasure-coded computation".The overarching goal of the project is to design and implement a solution for reducing the potential long latency in prediction-serving systems using erasure-coded computations. The project overcomes critical limitations in existing coded-computation solutions by employing machine learning in a novel way. Specifically, the project involves the following key tasks: (1) Designing a low-overhead coded-computation solution that efficiently supports complex non-linear functions by employing a learning-based approach; (2) Designing an efficient training methodology conforming to the constraints of real-world prediction-serving systems; and (3) System design and implementation.The software resulting from the project will be integrated into open-source prediction-serving systems for use in both research and education. Active engagement in promoting diversity in science, technology, engineering and mathematics disciplines and undergraduate mentorship will be continued. The findings from the project will be integrated into graduate and undergraduate courses in computer science.The results from the research project, including software, will be made available at http://www.cs.cmu.edu/~rvinayak/crii .This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.