Exploring a New Design Paradigm for Distributed Systems in Modern Data Centers

点击翻译按钮获取中文摘要

The end-to-end design “principle” pervades the design of virtually every modern distributed system. In its extreme form, critical functionality is implemented solely in end hosts, with a dumb but fast network to connect them. Unfortunately, such a network-oblivious approach is fundamentally inefficient, as it provides one-size-fits-all functionality to a multifarious set of applications and platforms.While this guideline has served us well in the past 30 years, technology evolution and tensions at the network interface suggest that it may be productive to revisit it. The goal of the proposed research program is to explore an alternative paradigm for designing distributed systems to leverage the capabilities of the emerging technology of software-defined networks (SDNs). The new paradigm will co-design distributed systems' logic and networking support to realize more efficient, scalable, and reliable systems. Achieving this vision requires a drastic shift in our approach to building, engineering, testing, securing, and deploying systems.I plan to establish a leading research group that will explore this new paradigm for designing a complete modern data center software/hardware stack, including building, testing, and deploying systems; exploring new security, management, accounting, and QoS techniques; and investigating new programming frameworks, domain-specific languages, and testing and debugging tools. This Discovery Grant funding is instrumental in bootstrapping this research group. The immediate research context is a network-integrated design for distributed storage systems, focusing on KV stores. To this end, the research program aims to (1) characterize the current KV systems to understand their network load and consistency properties; (2) design and prototype a network-integrated KV store; (3) explore designs for end-to-end QoS support; and (4) study the impact of departing from the end-to-end principle on security properties.The innovative paradigm we will explore has the potential to have a wide influence on technology, including many Canadian companies that build KV stores (e.g., NetApp, Ericsson, and IBM) and more than 150 Canadian data centers that use these systems. In particular, the program will result in four main contributions. First, the resulting designs will provide a blueprint for future systems and present novel solutions for stagnant QoS and scalability problems. Second, this program will build network-integrated designs for core distributed system mechanisms (e.g., QoS, security, consistency, and load balancing) that are applicable beyond KV stores. Third, this program will provide timely feedback on the suitability of the SDN API for enabling more efficient designs. Fourth, this program will provide an invaluable opportunity for training highly qualified personnel on cutting-edge networking and storage technology.