大规模数据中心多可用区高可靠纠删码技术的研究

在大规模数据中心中，与副本或备份相比，纠删码拥有更低的存储成本，从而得到了极其广泛的使用，如数据中心的多可用区中。然而，如何在多可用区中设计和部署低成本、高可靠、高性能、高扩展、易管理的纠删码，是当前云存储和大数据存储系统面临的重要挑战之一。..本项目从上述问题出发，针对多可用区的实际需求（如容忍单可用区故障、恢复低开销等），围绕时空多维的高可靠纠删码设计、基于可靠性分级的纠删码分组等关键科学问题，从低成本高可靠纠删码设计、普适的编解码加速方法、高速的数据分布策略、动态优先级的多级缓存算法、高效的I/O调度策略等五个方面开展研究，从存储设备层、分布式文件系统层、应用层多个层面进行具体实施，形成一整套大规模数据中心多可用区纠删码解决方案。重点突破低时延、低开销的高可靠纠删码，形成多种纠删码时空多维的融合，为产业界云存储和大数据存储系统的发展提供技术支撑。

In large scale data centers, compared to the replication and backup methods, erasure coding technique is widely used in data centers (e.g., multiple availability zones) due to its low storage cost. However, how to design and implement an erasure code with low cost, high reliability, high performance, high scalability and practical manageability, is one of the significant challenges for cloud storage and big data storage systems nowadays...To address the above-mentioned problems, in this project, firstly we focus on the real requirements of the multiple availability zones (i.e., tolerate single availability zone failure, low recovery overhead) and refine two major scientific issues, designing the temporal/spatial multi-dimensional high reliable erasure codes and classifying the erasure codes according the reliability levels. Secondly, we divide our research into five aspects, such as low cost and high reliable erasure coding design, pervasive encoding/decoding acceleration method, high speed data distribution strategy, dynamic priority multilevel cache algorithm, and efficient I/O scheduling approach. Finally, we implement the proposed design in three hierarchies (storage device, distributed file system and application levels), which aggregates a whole solution for multiple availability zones in large scale data centers. The key issue of this project is to breakthrough erasure coding design with low overhead and low latency, which can be organized as a temporal/spatial multi-dimensional erasure coding fusion. It can support the development of cloud storage and big data storage systems in industrial companies.