Exploring a New Design Paradigm for Distributed Systems in Modern Data Centers

端到端设计“原则”几乎渗透到每个现代分布式系统的设计中。在其极端形式中，关键功能仅在终端主机中实现，并使用哑但快速的网络连接它们。不幸的是，这种与网络无关的方法从根本上是低效的，因为它为各种各样的应用程序和平台提供了一刀切的功能。虽然这条准则在过去的30年里为我们提供了很好的服务，但技术的发展和网络接口的紧张局势表明，重新审视它可能会有成效。提出的研究计划的目标是探索设计分布式系统的另一种范例，以利用软件定义网络（sdn）的新兴技术的能力。新的范例将共同设计分布式系统的逻辑和网络支持，以实现更高效、可扩展和可靠的系统。实现这一愿景需要我们在构建、工程、测试、保护和部署系统的方法上进行重大转变。我计划建立一个领先的研究小组，探索设计一个完整的现代数据中心软件/硬件堆栈的新范式，包括构建、测试和部署系统；探索新的安全、管理、会计和QoS技术；研究新的编程框架、特定于领域的语言以及测试和调试工具。这项发现补助金的资助有助于引导这个研究小组。当前的研究背景是分布式存储系统的网络集成设计，重点是千伏存储。为此，研究计划旨在(1)表征当前KV系统，以了解其网络负载和一致性特性；(2)网络集成KV存储的设计与样机；(3)探索端到端QoS支持的设计；(4)研究偏离端到端原则对安全特性的影响。我们将探讨的创新模式有可能对技术产生广泛的影响，包括许多建立KV存储的加拿大公司（例如，NetApp，爱立信和IBM）和150多个使用这些系统的加拿大数据中心。特别是，该计划将产生四个主要贡献。首先，最终的设计将为未来的系统提供蓝图，并为停滞不前的QoS和可伸缩性问题提供新颖的解决方案。其次，该计划将为核心分布式系统机制（例如，QoS，安全性，一致性和负载平衡）构建网络集成设计，这些设计适用于KV存储之外。第三，该计划将及时提供关于SDN API适用性的反馈，以实现更高效的设计。第四，该计划将为培训尖端网络和存储技术的高素质人员提供宝贵的机会。

The end-to-end design “principle” pervades the design of virtually every modern distributed system. In its extreme form, critical functionality is implemented solely in end hosts, with a dumb but fast network to connect them. Unfortunately, such a network-oblivious approach is fundamentally inefficient, as it provides one-size-fits-all functionality to a multifarious set of applications and platforms.While this guideline has served us well in the past 30 years, technology evolution and tensions at the network interface suggest that it may be productive to revisit it. The goal of the proposed research program is to explore an alternative paradigm for designing distributed systems to leverage the capabilities of the emerging technology of software-defined networks (SDNs). The new paradigm will co-design distributed systems' logic and networking support to realize more efficient, scalable, and reliable systems. Achieving this vision requires a drastic shift in our approach to building, engineering, testing, securing, and deploying systems.I plan to establish a leading research group that will explore this new paradigm for designing a complete modern data center software/hardware stack, including building, testing, and deploying systems; exploring new security, management, accounting, and QoS techniques; and investigating new programming frameworks, domain-specific languages, and testing and debugging tools. This Discovery Grant funding is instrumental in bootstrapping this research group. The immediate research context is a network-integrated design for distributed storage systems, focusing on KV stores. To this end, the research program aims to (1) characterize the current KV systems to understand their network load and consistency properties; (2) design and prototype a network-integrated KV store; (3) explore designs for end-to-end QoS support; and (4) study the impact of departing from the end-to-end principle on security properties.The innovative paradigm we will explore has the potential to have a wide influence on technology, including many Canadian companies that build KV stores (e.g., NetApp, Ericsson, and IBM) and more than 150 Canadian data centers that use these systems. In particular, the program will result in four main contributions. First, the resulting designs will provide a blueprint for future systems and present novel solutions for stagnant QoS and scalability problems. Second, this program will build network-integrated designs for core distributed system mechanisms (e.g., QoS, security, consistency, and load balancing) that are applicable beyond KV stores. Third, this program will provide timely feedback on the suitability of the SDN API for enabling more efficient designs. Fourth, this program will provide an invaluable opportunity for training highly qualified personnel on cutting-edge networking and storage technology.