CAREER: Rethinking Replication in Highly Available and Reliable Data Stores

该项目的目标是建立高度可靠和可用的分布式存储系统，这是大规模互联网服务的关键组成部分的长期和日益普遍的挑战。该提案旨在研究和解决当前分布式存储系统设计中的不一致性，从几个重要方面，包括其抽象，性能，通信和故障模型。这些不一致性通常源于分布式系统中的传统设计，这些设计没有为今天的互联网规模做好准备。这项研究可能会对工业和社会产生重大影响，因为分布式系统是云计算、无服务器计算和高性能计算等现代计算基础设施的基石。特别是，这项工作将与微软、谷歌、MongoDB和Cockroach中广泛使用的分布式存储系统合作完成。PI正在与他们的部门合作，在云计算，分布式系统，可靠系统和软件工程的一般领域提供多学科课程。PI将把本提案中的主题融入到他所教授的课程中。传统上，系统通常使用一种称为状态机复制（SMR）的方法来实现可靠性。SMR通常被视为一个独立的模块，有一个明确的边界，如键-值接口将其与系统的其余部分隔离开来。然而，这在今天的系统中产生了问题。例如，在地理复制系统中使用SMR作为黑盒可能会导致很大的性能损失，因为每个应用程序请求可能会转换为多个SMR操作，并且每个操作至少需要广域往返才能完成。该建议从实践中的不同方面总结了现有制度的四个障碍（抽象、性能、通信模型和故障模型），并针对这些障碍提出了四个研究重点：1）使用事务数据结构来丰富应用程序和数据存储之间的抽象层，2）研究利用多核资源来提高地理复制的性能，3）在一个比现状更广义的通信模型下研究先进的SMR技术; 4）研究容忍一种新的故障模型，沉默的数据损坏。拟议的工作将开发一套新颖的变革性技术，包括一个新的复制事务数据结构库，支持构建一般类型的应用程序，其他新的系统设计和实施，以提供快速和一致的复制，将封装在最先进的研究产品，并将大大提高性能和故障，该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project targets long-lasting and increasingly pervasive challenges in building highly reliable and available distributed storage systems, which are the key components in large-scale Internet services. The proposal aims to study and address insufficiencies in current distributed storage system designs from several important aspects including their abstraction, performance, communication, and failure models. These insufficiencies typically arise from the conventional designs in distributed systems that were not prepared for today’s Internet scale. This research may produce a major impact on industry and society because distributed systems are the cornerstones of modern computing infrastructures such as cloud computing, serverless computing, and high-performance computing. In particular, this work will be done in collaboration with widely used distributed storage systems built in Microsoft, Google, MongoDB, and Cockroach. The PI is working with their department to broaden the course offerings with multidisciplinary courses in the general area of cloud computing, distributed systems, reliable systems, and software engineering. The PI will incorporate the topics in this proposal into the courses he is teaching.Traditionally, systems often use a method called state machine replication (SMR) to achieve reliability. SMR is often treated as an independent module, with a clear boundary such as a key-value interface isolating it from the rest of the system. However, this creates problems in today’s systems. For example, using SMR as a blackbox in geo-replicated systems could incur a great performance penalty because each application request could translate into multiple SMR operations and each operation takes at least a wide-area round-trip to finish. This proposal summarizes four impediments of existing systems from different aspects in practice (abstraction, performance, communication model, and failure model), and proposes four research thrusts targeting these impediments: 1) use transactional data structures to enrich the abstraction layer between applications and the data store, 2) investigate leveraging multicore resources to improve performance for geo-replication, 3) study advanced SMR techniques under a more generalized communication model than the status quo, 4) study tolerating a new failure model, silent data corruption. The proposed work will develop a set of novel, transformative technologies, including a new replicated transactional data structure library that supports building general types of applications, other new system designs and implementations for providing fast and consistent replication that will be encapsulated in state-of-the-art research products and will dramatically improve the performance and fault-tolerance of modern distributed systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.