CAREER: Towards Scalable Datacenter Services with Strong Robustness Guarantees

现代数据中心已经达到数十万台服务器，现代分布式服务可以分布在全球多个甚至数十个数据中心。为了提高可扩展性、可用性和性能，这些服务越来越多地采用数据一致性的弱化模型。事实证明，对于Web爬行、搜索和内容分发等应用程序，这种权衡非常成功。另一方面，最近将更动态的应用程序转移到“云”的趋势预示着服务需求的转变-其中丢失数据或无序应用操作可能是不可接受的-以及对数据和服务的并发访问的广泛需求。为了满足这些不断变化的需求，该研究项目重新考虑了构建具有强大健壮性保证和规模化的存储和复制系统的挑战。认识到大型、复杂的系统通常是从较小的组建立起来的-它们要么提供不同的功能，要么将一些较大的状态空间划分为更小、更易于管理的部分-它利用组组成方法来解决问题。首先，这项研究为较小的节点组开发了新的协议，这些协议以最小的开销提供强大的性能。其次，提出了一种协调服务和一套自适应地组织这些组并将它们组合在一起的管理算法。这些算法探索了动态负载平衡、拓扑控制和安全等问题。这项研究包括大量的系统建设，包括将其核心服务整合到其他几个分布式系统中(例如，可扩展的虚拟世界和分布式名称解析服务)。非常丰富和多样化的互联网服务从数据中心运行出来，构成了当今人们在线体验的核心。这项研究开发的算法和软件系统可能会降低开发可扩展、可靠、安全、高效和易于管理的数据中心服务的障碍。通过降低构建强大的大规模应用程序的技术难度-通过为它们提供坚实的基础-这项研究可能会使互联网服务进一步创新。

Modern datacenters have reached hundreds of thousands of servers, and modern distributed services can be spread over multiple or even dozens of datacenters worldwide. For scalability, availability, and performance, these services have increasingly embraced a weakened model of data consistency. This trade-off proved highly successful for applications such as web crawling, search, and content distribution. On the other hand, the recent trend to move ever-more dynamic applications to the "cloud" portends a shift in service requirements---in which losing data or applying operations out-of-order may not be acceptable---as does far-flung demand for concurrent access to data and services.To meet these changing needs, this research project reconsiders the challenge of building storage and replicated systems with strong robustness guarantees and at scale. Recognizing that large-scale, complex systems are typically built-up from smaller groups---which either provide different functionality or partition some larger state-space into smaller, more manageable parts---it leverages a group compositional approach to tackle the problem. First, the research develops novel protocols for smaller groups of nodes that offer strong properties at minimal overhead. Second, it proposes a coordination service and a suite of management algorithms that adaptively organizes these groups and composes them together. These algorithms explore problems of dynamic load balancing, topological control, and security. The research includes substantial systems building, including the incorporation of its core services into several other distributed systems (e.g., a scalable virtual world and a istributed name resolution service).The remarkably rich and varied Internet services run out of datacenters form the core of people's online experiences today. The algorithms and software systems developed by this research may lower the barrier to developing datacenter services that are scalable, reliable, secure, efficient, and easy to manage. And by reducing the technical difficulty of building robust, large-scale applications---by providing them with a firm foundation---this research may enable further innovation in Internet services.