Parity models: erasure-coded resilience for prediction serving systems
Parity models: erasure-coded resilience for prediction serving systems
复制标题
DOI：
10.1145/3341301.3359654
复制
发表时间：
2019-10
期刊：
影响因子：
--
通讯作者：
J. Kosaian;K. V. Rashmi;S. Venkataraman
中科院分区：
文献类型：
--
作者：
J. Kosaian;K. V. Rashmi;S. Venkataraman
机器学习模型正在成为许多应用程序的主要工作。服务通过预测服务系统部署模型，预测服务系统接受查询并通过对模型执行推理来返回预测。预测服务系统通常在集群环境中的许多机器上运行，因此容易出现减速和故障，从而增加尾部延迟。擦除编码是一种流行的技术，用于在存储和通信系统中实现对数据不可用性的资源有效的弹性。然而，现有的方法，赋予擦除编码的弹性分布式计算只适用于一个严重有限的功能，排除了他们的使用许多服务的工作负载，如神经网络推理。我们引入奇偶校验模型，一种新的方法，使预测服务系统中的擦除编码的弹性。奇偶校验模型是一种神经网络，经过训练可以将擦除编码的查询转换为一种形式，使解码器能够重建缓慢或失败的预测。我们在ParM中实现奇偶校验模型，ParM是一个预测服务系统，它利用了擦除编码的弹性。ParM将多个查询编码为“奇偶查询”，使用奇偶模型对奇偶查询执行推断，并通过使用奇偶模型的输出来解码不可用预测的近似。我们展示了奇偶模型的适用性，图像分类，语音识别和对象定位任务。使用奇偶校验模型，ParM将第99.9百分位数和中位数延迟之间的差距减少了3.5倍，同时保持相同的中位数。这些结果显示了奇偶校验模型的潜力，以解锁一个新的途径，赋予资源有效的预测服务系统的弹性。
Machine learning models are becoming the primary work-horses for many applications. Services deploy models through prediction serving systems that take in queries and return predictions by performing inference on models. Prediction serving systems are commonly run on many machines in cluster settings, and thus are prone to slowdowns and failures that inflate tail latency. Erasure coding is a popular technique for achieving resource-efficient resilience to data unavailability in storage and communication systems. However, existing approaches for imparting erasure-coded resilience to distributed computation apply only to a severely limited class of functions, precluding their use for many serving workloads, such as neural network inference. We introduce parity models, a new approach for enabling erasure-coded resilience in prediction serving systems. A parity model is a neural network trained to transform erasure-coded queries into a form that enables a decoder to reconstruct slow or failed predictions. We implement parity models in ParM, a prediction serving system that makes use of erasure-coded resilience. ParM encodes multiple queries into a "parity query," performs inference over parity queries using parity models, and decodes approximations of unavailable predictions by using the output of a parity model. We showcase the applicability of parity models to image classification, speech recognition, and object localization tasks. Using parity models, ParM reduces the gap between 99.9th percentile and median latency by up to 3.5X, while maintaining the same median. These results display the potential of parity models to unlock a new avenue to imparting resource-efficient resilience to prediction serving systems.