NeTS: Small: Collaborative Research: Rethinking Erasure Codes for Cloud Storage: A Quantitative Framework for Latency, Reliability, and Cost Optimization

该项目旨在开发一个分析框架，通过研究新的调度和修复策略，量化擦除编码存储的尾部延迟和可靠性，要求重新思考在线存储的擦除编码。随着大规模存储系统（如Microsoft Azure和Facebook）越来越多地采用擦除编码，主要依赖于系统设计启发式的传统方法在延迟和可靠性优化方面已经不足以推动性能界限。采用动态工作负载调度和在线修复的擦除编码存储的尾部延迟和可靠性量化是一个有待解决的问题。通过关键权衡和相关工程“控制旋钮”的数学结晶来阐明设计空间的工作很少。该项目将实现延迟、可靠性和存储成本的联合优化，这要求重新考虑擦除编码存储优化和服务定价模型。本提案将集中在以下具体方面：(1)研究一系列新的概率调度策略，并扩展顺序统计分析，以获得具有任意配置，一般服务时间分布和潜在差异化服务类别的擦除编码存储的尾部延迟的封闭形式界限。(2)通过一种新的可靠性度量，即数据丢失时间，本研究将研究在线维修策略，该策略使用一类带宽效率高的代码来显著提高可靠性，从而在维修及时性和可靠性优化之间实现权衡。(3)通过理论分析，在擦除码选择、块放置、网络资源分配、缓存管理、动态调度、定价和在线修复策略等7个关键控制维度上，寻求可靠性、延迟和存储成本共同优化的整体解决方案。(4)将提出的框架与现有的云系统进行整合，弥合理论结果与实际系统之间的差距。通过开发新的分析模型和算法来联合优化延迟、可靠性和存储成本，该项目将要求重新思考擦除编码存储设计和服务定价模型。它将为这些理论和系统的教学提供新颖的跨学科课程模块。

This project aims to develop an analytical framework that quantifies tail latency and reliability of erasure-coded storage through investigation of novel scheduling and repair strategies, mandating rethinking of erasure codes for online storage. As erasure coding is increasingly adopted by large-scale storage systems such as Microsoft Azure and Facebook, conventional approaches that primarily rely on system design heuristics has become inadequate in pushing the performance boundaries in terms of latency and reliability optimization. Quantifying tail latency and reliability of an erasure-coded storage that employs dynamic workload scheduling and online repair is an open problem. There exists little work illuminating the design space via mathematical crystallization of key tradeoffs and associated engineering "control knobs". This project will enable a joint optimization of latency, reliability and storage cost, which mandates rethinking of erasure-coded storage optimization and service pricing models. This proposal will concentrate on the following specific aspects: (1) Investigating a family of new probabilistic scheduling policies and extend order statistic analysis to derive closed-form bounds on tail latency for erasure-coded storage with arbitrary configurations, general service-time distributions, and potentially differentiated service classes. (2) Through a novel reliability metric, Time to Data Loss, The research will investigate online repair strategies that significantly improve reliability using a class of bandwidth-efficient codes, enabling a tradeoff between repair timeliness and reliability optimization. (3) Employ the theoretical analysis in this research to pursue a holistic solution that jointly optimizes reliability, latency, and storage costs over seven key control dimensions: choice of erasure codes, chunk placement, network resource allocation, cache management, dynamic scheduling, pricing, and online repair strategy. (4) Integrate the proposed framework with current cloud systems to bridge the gap between the theoretical results and practical systems. By developing new analytical models and algorithms for joint optimization of latency, reliability, and storage cost, the project will mandate rethinking of erasure-coded storage design and service pricing models. It will produces novel, interdisciplinary curriculum modules for teaching both these theories and systems.