NeTS: Small: Collaborative Research: Rethinking Erasure Codes for Cloud Storage: A Quantitative Framework for Latency, Reliability, and Cost Optimization

该项目旨在开发一个分析框架，通过研究新的调度和修复策略，量化擦除编码存储的尾部延迟和可靠性，并要求重新思考在线存储的擦除码。随着擦除编码越来越多地被诸如Microsoft Azure和Facebook之类的大规模存储系统采用，主要依赖于系统设计的传统方法在延迟和可靠性优化方面已经不足以推动性能边界。采用动态工作负载调度和在线修复的擦除编码存储的量化尾部延迟和可靠性是一个开放的问题。通过关键权衡和相关工程“控制旋钮”的数学结晶来阐明设计空间的工作很少。该项目将实现延迟、可靠性和存储成本的联合优化，这要求重新思考擦除编码存储优化和服务定价模型。该建议将集中在以下具体方面：（1）调查一个家庭的新的概率调度策略和扩展顺序统计分析，以获得封闭形式的边界上的尾延迟的擦除编码存储与任意配置，一般的服务时间分布，和潜在的差异服务类。(2)通过一种新的可靠性指标，数据丢失时间，研究将调查在线修复策略，显着提高可靠性使用一类带宽效率的代码，使修复及时性和可靠性优化之间的权衡。(3)采用本研究中的理论分析来寻求一种整体解决方案，该解决方案在七个关键控制维度上联合优化可靠性，延迟和存储成本：擦除码的选择，块放置，网络资源分配，缓存管理，动态调度，定价和在线修复策略。(4)将所提出的框架与当前的云系统集成，以弥合理论结果与实际系统之间的差距。通过开发新的分析模型和算法来联合优化延迟、可靠性和存储成本，该项目将要求重新思考擦除编码存储设计和服务定价模型。它将产生新的，跨学科的课程模块，用于教授这些理论和系统。

This project aims to develop an analytical framework that quantifies tail latency and reliability of erasure-coded storage through investigation of novel scheduling and repair strategies, mandating rethinking of erasure codes for online storage. As erasure coding is increasingly adopted by large-scale storage systems such as Microsoft Azure and Facebook, conventional approaches that primarily rely on system design heuristics has become inadequate in pushing the performance boundaries in terms of latency and reliability optimization. Quantifying tail latency and reliability of an erasure-coded storage that employs dynamic workload scheduling and online repair is an open problem. There exists little work illuminating the design space via mathematical crystallization of key tradeoffs and associated engineering "control knobs". This project will enable a joint optimization of latency, reliability and storage cost, which mandates rethinking of erasure-coded storage optimization and service pricing models. This proposal will concentrate on the following specific aspects: (1) Investigating a family of new probabilistic scheduling policies and extend order statistic analysis to derive closed-form bounds on tail latency for erasure-coded storage with arbitrary configurations, general service-time distributions, and potentially differentiated service classes. (2) Through a novel reliability metric, Time to Data Loss, The research will investigate online repair strategies that significantly improve reliability using a class of bandwidth-efficient codes, enabling a tradeoff between repair timeliness and reliability optimization. (3) Employ the theoretical analysis in this research to pursue a holistic solution that jointly optimizes reliability, latency, and storage costs over seven key control dimensions: choice of erasure codes, chunk placement, network resource allocation, cache management, dynamic scheduling, pricing, and online repair strategy. (4) Integrate the proposed framework with current cloud systems to bridge the gap between the theoretical results and practical systems. By developing new analytical models and algorithms for joint optimization of latency, reliability, and storage cost, the project will mandate rethinking of erasure-coded storage design and service pricing models. It will produces novel, interdisciplinary curriculum modules for teaching both these theories and systems.