Exploring a New Design Paradigm for Distributed Systems in Modern Data Centers

端到端设计“原则”几乎渗透到每个现代分布式系统的设计中。在其极端形式中，关键功能仅在终端主机中实施，并使用一个愚蠢但快速的网络将它们连接起来。不幸的是，这种无视网络的方法从根本上来说是低效的，因为它为各种应用程序和平台提供了一刀切的功能。*虽然这一指导方针在过去30年里一直很好地服务于我们，但技术发展和网络接口的紧张局势表明，重新使用它可能是有成效的。拟议研究计划的目标是探索另一种设计分布式系统的范例，以利用软件定义网络(SDN)这一新兴技术的能力。新的范例将共同设计分布式系统的逻辑和网络支持，以实现更高效、可扩展和可靠的系统。实现这一愿景需要我们在构建、工程、测试、保护和部署系统方面的方法发生重大转变。*我计划建立一个领先的研究小组，探索这一新范式，以设计完整的现代数据中心软件/硬件堆栈，包括构建、测试和部署系统；探索新的安全、管理、会计和服务质量技术；以及研究新的编程框架、特定于领域的语言以及测试和调试工具。这笔探索基金对这个研究小组的自力更生起到了重要作用。*当前的研究背景是分布式存储系统的网络集成设计，重点是KV商店。为此，研究计划旨在(1)描述当前KV系统的特征，以了解其网络负载和一致性属性；(2)设计和制作网络集成KV商店的原型；(3)探索端到端服务质量支持的设计；以及(4)研究偏离端到端原则对安全属性的影响。*我们将探索的创新范式有可能对技术产生广泛影响，包括许多构建KV商店的加拿大公司(如NetApp、爱立信和IBM)和150多个使用这些系统的加拿大数据中心。特别是，该计划将产生四个主要贡献。首先，由此产生的设计将为未来的系统提供蓝图，并为停滞不前的服务质量和可伸缩性问题提供新的解决方案。其次，该计划将为适用于KV门店以外的核心分布式系统机制(例如，服务质量、安全性、一致性和负载平衡)构建网络集成设计。第三，该计划将就SDN API是否适合实现更高效的设计提供及时的反馈。第四，该计划将为培训尖端网络和存储技术方面的高素质人员提供宝贵的机会。

The end-to-end design “principle” pervades the design of virtually every modern distributed system. In its extreme form, critical functionality is implemented solely in end hosts, with a dumb but fast network to connect them. Unfortunately, such a network-oblivious approach is fundamentally inefficient, as it provides one-size-fits-all functionality to a multifarious set of applications and platforms.******While this guideline has served us well in the past 30 years, technology evolution and tensions at the network interface suggest that it may be productive to revisit it. The goal of the proposed research program is to explore an alternative paradigm for designing distributed systems to leverage the capabilities of the emerging technology of software-defined networks (SDNs). The new paradigm will co-design distributed systems' logic and networking support to realize more efficient, scalable, and reliable systems. Achieving this vision requires a drastic shift in our approach to building, engineering, testing, securing, and deploying systems.******I plan to establish a leading research group that will explore this new paradigm for designing a complete modern data center software/hardware stack, including building, testing, and deploying systems; exploring new security, management, accounting, and QoS techniques; and investigating new programming frameworks, domain-specific languages, and testing and debugging tools. This Discovery Grant funding is instrumental in bootstrapping this research group. ******The immediate research context is a network-integrated design for distributed storage systems, focusing on KV stores. To this end, the research program aims to (1) characterize the current KV systems to understand their network load and consistency properties; (2) design and prototype a network-integrated KV store; (3) explore designs for end-to-end QoS support; and (4) study the impact of departing from the end-to-end principle on security properties.******The innovative paradigm we will explore has the potential to have a wide influence on technology, including many Canadian companies that build KV stores (e.g., NetApp, Ericsson, and IBM) and more than 150 Canadian data centers that use these systems. In particular, the program will result in four main contributions. First, the resulting designs will provide a blueprint for future systems and present novel solutions for stagnant QoS and scalability problems. Second, this program will build network-integrated designs for core distributed system mechanisms (e.g., QoS, security, consistency, and load balancing) that are applicable beyond KV stores. Third, this program will provide timely feedback on the suitability of the SDN API for enabling more efficient designs. Fourth, this program will provide an invaluable opportunity for training highly qualified personnel on cutting-edge networking and storage technology.