NeTS: Small: Collaborative Research: Rethinking Erasure Codes for Cloud Storage: A Quantitative Framework for Latency, Reliability, and Cost Optimization

NeTS：小型：协作研究：重新思考云存储纠删码：延迟、可靠性和成本优化的定量框架

• 批准号: 1618335
• 负责人: Vaneet Aggarwal
• 金额: $ 25万
• 依托单位: Purdue University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-10-01 至 2022-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1618335&HistoricalAwards=false
• 关键词: NeTS; Small; Collaborative; Research; Rethinking

This project aims to develop an analytical framework that quantifies tail latency and reliability of erasure-coded storage through investigation of novel scheduling and repair strategies, mandating rethinking of erasure codes for online storage. As erasure coding is increasingly adopted by large-scale storage systems such as Microsoft Azure and Facebook, conventional approaches that primarily rely on system design heuristics has become inadequate in pushing the performance boundaries in terms of latency and reliability optimization. Quantifying tail latency and reliability of an erasure-coded storage that employs dynamic workload scheduling and online repair is an open problem. There exists little work illuminating the design space via mathematical crystallization of key tradeoffs and associated engineering "control knobs". This project will enable a joint optimization of latency, reliability and storage cost, which mandates rethinking of erasure-coded storage optimization and service pricing models. This proposal will concentrate on the following specific aspects: (1) Investigating a family of new probabilistic scheduling policies and extend order statistic analysis to derive closed-form bounds on tail latency for erasure-coded storage with arbitrary configurations, general service-time distributions, and potentially differentiated service classes. (2) Through a novel reliability metric, Time to Data Loss, The research will investigate online repair strategies that significantly improve reliability using a class of bandwidth-efficient codes, enabling a tradeoff between repair timeliness and reliability optimization. (3) Employ the theoretical analysis in this research to pursue a holistic solution that jointly optimizes reliability, latency, and storage costs over seven key control dimensions: choice of erasure codes, chunk placement, network resource allocation, cache management, dynamic scheduling, pricing, and online repair strategy. (4) Integrate the proposed framework with current cloud systems to bridge the gap between the theoretical results and practical systems. By developing new analytical models and algorithms for joint optimization of latency, reliability, and storage cost, the project will mandate rethinking of erasure-coded storage design and service pricing models. It will produces novel, interdisciplinary curriculum modules for teaching both these theories and systems.

该项目旨在开发一个分析框架，该框架通过调查新颖的调度和维修策略来量化擦除编码存储的尾巴潜伏期和可靠性，从而强制重新考虑擦除擦除代码以进行在线存储。由于诸如Microsoft Azure和Facebook之类的大规模存储系统越来越多地采用了擦除编码，因此主要依赖系统设计启发式方法的常规方法已经不足以在延迟和可靠性优化方面突破性能界限。使用动态工作负载调度和在线维修的擦除编码存储的尾巴潜伏期和可靠性是一个开放的问题。通过数学结晶和关联的工程“控制旋钮”，几乎没有工作来照亮设计空间。该项目将实现延迟，可靠性和存储成本的联合优化，这要求重新思考擦除编码的存储优化和服务定价模型。该提案将集中在以下特定方面：（1）调查一个新的概率调度策略家庭，并扩展订单统计分析，以在擦除层潜伏期的封闭形式范围内，以使用任意配置，一般服务时间分布以及潜在差异化的服务类别的擦除编码存储。 （2）通过新颖的可靠性指标，数据丢失的时间，研究将调查在线维修策略，这些维修策略通过一类带宽效率的代码可显着提高可靠性，从而在维修及时性和可靠性优化之间进行权衡。 （3）在本研究中采用理论分析来追求一个整体解决方案，该解决方案共同优化了七个关键控制维度的可靠性，延迟和存储成本：选择擦除代码，块位置，网络资源分配，缓存管理，动态调度，定价，定价和在线维修策略。 （4）将提出的框架与当前的云系统集成在一起，以弥合理论结果与实际系统之间的差距。通过开发新的分析模型和算法，以优化延迟，可靠性和存储成本的联合优化，该项目将要求重新思考擦除编码的存储设计和服务定价模型。它将生成新颖的跨学科课程模块，用于教授这些理论和系统。