NeTS: Small: Collaborative Research: Rethinking Erasure Codes for Cloud Storage: A Quantitative Framework for Latency, Reliability, and Cost Optimization

该项目旨在开发一个分析框架，通过研究新的调度和修复策略来量化擦除编码存储的尾部延迟和可靠性，要求重新考虑在线存储的擦除编码。随着擦除编码越来越多地被Microsoft Azure和Facebook等大规模存储系统采用，主要依赖系统设计试探法的传统方法在延迟和可靠性优化方面已经不能满足性能要求。量化采用动态工作负载调度和在线修复的擦除编码存储的尾部延迟和可靠性是一个悬而未决的问题。通过关键权衡和相关工程“控制旋钮”的数学结晶来照亮设计空间的工作很少。该项目将实现延迟、可靠性和存储成本的联合优化，这要求重新考虑擦除编码存储优化和服务定价模型。该方案将集中在以下几个方面：(1)研究一类新的概率调度策略，并扩展顺序统计分析，推导出任意配置、一般服务时间分布和潜在区分服务类别的擦除编码存储的尾部延迟的闭合形式的界。(2)通过一种新的可靠性度量-数据丢失时间，研究在线修复策略，该策略使用一类带宽高效码来显著提高可靠性，从而在修复及时性和可靠性优化之间进行权衡。(3)利用本研究中的理论分析，寻求在七个关键控制维度上联合优化可靠性、延迟和存储成本的整体解决方案：擦除码选择、块放置、网络资源分配、缓存管理、动态调度、定价和在线修复策略。(4)将所提出的框架与现有的云系统相结合，以弥补理论结果与实际系统之间的差距。通过开发用于联合优化延迟、可靠性和存储成本的新分析模型和算法，该项目将要求重新考虑擦除编码存储设计和服务定价模型。它将产生新颖的、跨学科的课程模块，用于教授这些理论和系统。

This project aims to develop an analytical framework that quantifies tail latency and reliability of erasure-coded storage through investigation of novel scheduling and repair strategies, mandating rethinking of erasure codes for online storage. As erasure coding is increasingly adopted by large-scale storage systems such as Microsoft Azure and Facebook, conventional approaches that primarily rely on system design heuristics has become inadequate in pushing the performance boundaries in terms of latency and reliability optimization. Quantifying tail latency and reliability of an erasure-coded storage that employs dynamic workload scheduling and online repair is an open problem. There exists little work illuminating the design space via mathematical crystallization of key tradeoffs and associated engineering "control knobs".This project will enable a joint optimization of latency, reliability and storage cost, which mandates rethinking of erasure-coded storage optimization and service pricing models. This proposal will concentrate on the following specific aspects: (1) Investigating a family of new probabilistic scheduling policies and extend order statistic analysis to derive closed-form bounds on tail latency for erasure-coded storage with arbitrary configurations, general service-time distributions, and potentially differentiated service classes. (2) Through a novel reliability metric, Time to Data Loss, The research will investigate online repair strategies that significantly improve reliability using a class of bandwidth-efficient codes, enabling a tradeoff between repair timeliness and reliability optimization. (3) Employ the theoretical analysis in this research to pursue a holistic solution that jointly optimizes reliability, latency, and storage costs over seven key control dimensions: choice of erasure codes, chunk placement, network resource allocation, cache management, dynamic scheduling, pricing, and online repair strategy. (4) Integrate the proposed framework with current cloud systems to bridge the gap between the theoretical results and practical systems. By developing new analytical models and algorithms for joint optimization of latency, reliability, and storage cost, the project will mandate rethinking of erasure-coded storage design and service pricing models. It will produces novel, interdisciplinary curriculum modules for teaching both these theories and systems.