CRII: CSR: Mitigating Tail Latency in Prediction-serving Systems using Coding-theoretic Tools

预测服务系统接受查询并从机器学习模型返回预测。例如，图像分类服务将把图像作为查询，并预测图像中是否有汽车。预测服务系统在当今各种各样的应用中越来越重要。这些系统通常在大型计算基础设施中运行，其中故障和减速是常见的。这种不可用性会导致查询-响应延迟的显著增加，从而降低服务质量。该研究项目使用编码理论领域的工具“擦除编码计算”，设计并实现了预测服务系统鲁棒性的资源高效解决方案。该项目的总体目标是设计和实现一个解决方案，以减少使用擦除编码计算的预测服务系统中潜在的长延迟。该项目以一种新颖的方式利用机器学习，克服了现有编码计算解决方案的关键限制。具体而言，该项目涉及以下关键任务：(1)采用基于学习的方法设计一种低开销的编码计算解决方案，有效地支持复杂的非线性函数；(2)设计符合现实世界预测服务系统约束的高效训练方法；(3)系统设计与实现。该项目产生的软件将集成到开源预测服务系统中，用于研究和教育。将继续积极参与促进科学、技术、工程和数学学科的多样性和本科生指导。该项目的研究结果将整合到计算机科学的研究生和本科生课程中。研究项目的结果，包括软件，将在http://www.cs.cmu.edu/~rvinayak/crii上提供。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Prediction-serving systems take in queries and return predictions from a machine learning model. For example, an image classification service would take an image as a query and respond with a prediction of whether there is a car in the image. Prediction-serving systems are increasingly important for a wide variety of applications today. These systems are typically run in large-scale computing infrastructures, where failures and slowdowns are common. Such unavailability results in significant increase in the query-response latency, thereby degrading the quality-of-service. The research project designs and implements a resource-efficient solution for robustness in prediction-serving systems using a tool from the domain of coding theory called "erasure-coded computation".The overarching goal of the project is to design and implement a solution for reducing the potential long latency in prediction-serving systems using erasure-coded computations. The project overcomes critical limitations in existing coded-computation solutions by employing machine learning in a novel way. Specifically, the project involves the following key tasks: (1) Designing a low-overhead coded-computation solution that efficiently supports complex non-linear functions by employing a learning-based approach; (2) Designing an efficient training methodology conforming to the constraints of real-world prediction-serving systems; and (3) System design and implementation.The software resulting from the project will be integrated into open-source prediction-serving systems for use in both research and education. Active engagement in promoting diversity in science, technology, engineering and mathematics disciplines and undergraduate mentorship will be continued. The findings from the project will be integrated into graduate and undergraduate courses in computer science.The results from the research project, including software, will be made available at http://www.cs.cmu.edu/~rvinayak/crii .This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.