CSR: Small: Scalable Coordination for Wide-Area Distributed Systems

并发执行任务的可扩展协调是大规模分布式系统中一个具有挑战性的开放问题。对于需要很少或不需要通信的松散依赖任务，简单的抽象（如屏障同步）足以进行协调。然而，对于需要更严格同步的大规模云计算和Web服务应用程序（例如在线事务处理系统，分布式文件系统和图形处理应用程序），需要细粒度的复杂协调服务。特别是随着对用于电子商务、社交网络和物联网的大规模网络服务的需求的增加，广域（即，传统的分布式协调技术无法扩展到广域网以支持这些新一代应用。集中式协调无法扩展相对于广域中增加的距离，而分布式协调无法扩展相对于所涉及的节点的数量。该研究项目声称，使用一种名为Maestro的新型混合设计，可以在广域范围内实现分布式任务的可扩展协调。Maestro框架将解决以下研究问题：（1）完全集中和完全分散的协调解决方案的局限性是什么？混合分层方法的可扩展性优势是什么？(2)如何利用位置感知来实现跨广域的高性能？(3)如何利用分区感知来实现广域一致性？Maestro框架将采用分层锁代理架构，具有新颖的锁租赁机制和智能/自适应锁迁移。这种组合允许控制的灵活性，并提供集中和分散方法的最佳效果。作为各自域的权威，代理在运行时学习和适应任务的访问模式，以提高锁的本地性和可扩展性，同时它们也具有自治权，允许独立的任务以分散的方式启动和执行。Maestro将提供优化，例如在请求锁之前主动将锁租赁给服务器，（改变锁的主站点分配），以及共享/部分锁租赁，以在适当的时候选择性地允许分散协调和宽松的一致性。建议的Maestro框架将在两个流行的分布式应用领域进行评估：广域ZooKeeper用于分布式协调和广域分布式元数据管理。Maestro将填补紧密耦合的一致性关键分布式应用程序的广域可扩展协调方面的重要空白，并将通过研究生和本科生课程开发，通过组织学术研讨会，向K-12学生推广，招募少数群体以及向社区分发工具和软件来增强科学/技术理解，从而实现更广泛的影响。

Scalable coordination of concurrently executing tasks is a challenging open problem for large-scale distributed systems. For loosely dependent tasks that require little or no communication, simple abstractions such as barrier synchronization suffice for coordination. However, for large-scale cloud-computing and web-services applications that require tighter synchronization (such as online transaction processing systems, distributed file systems, and graph processing applications), a fine-grained complex coordination service is needed. Especially with increasing demand for large-scale web-services for e-commerce, social networking, and Internet of Things, the coordination of tasks over wide-area (i.e., across clusters, across datacenters, and across Internet) has recently gained greater importance.Traditional distributed coordination techniques fail to scale for wide-area networks to support these new generation applications. Centralized coordination fails to scale with respect to the increased distances in the wide-area, whereas distributed coordination fails to scale with respect to the number of nodes involved. This research project claims that it is possible to achieve scalable coordination of distributed tasks over wide-area using a novel hybrid design, called Maestro. The Maestro framework will address the following research questions: (1) What are the limits of fully-centralized and fully-decentralized solutions to coordination, and what are the scalability benefits of a hybrid hierarchical approach? (2) How can locality-awareness be utilized to achieve high-performance across wide-area? (3) How can partition-awareness be utilized to achieve consistency across wide-area? The Maestro framework will employ a hierarchical lock broker architecture with a novel lock-leasing mechanism and smart/adaptive lock migration. This combination allows flexibility of control and provides the best of both centralized and decentralized approaches. As the authority of their respective domains, the brokers learn and adapt to the access patterns of tasks at runtime to improve lock-locality and hence scalability, while they also have autonomy to allow independent tasks to be initiated and executed in a decentralized manner. Maestro will provide optimizations such as proactive leasing of locks to servers even before they are requested, lock migration (changing primary site assignment of locks), and shared/fractional lock-leasing to selectively allow decentralized coordination and relaxed consistency when appropriate.The proposed Maestro framework will be evaluated on two popular distributed application domains: wide-area ZooKeeper for distributed coordination, and wide-area distributed metadata management. Maestro will fill an important gap in the wide-area scalable coordination of tightly-coupled consistency-critical distributed applications, and it will enable further broader impacts through graduate and undergraduate level curriculum development, enhancing scientific/technological understanding via organizing academic workshops, outreaching to K-12 students, recruitment of minority groups, and distributing tools and software to the community.