Exploring a New Design Paradigm for Distributed Systems in Modern Data Centers

端到端设计“原则”几乎渗透到每一个现代分布式系统的设计中。在其极端形式中，关键功能仅在终端主机中实现，并通过一个愚蠢但快速的网络连接它们。不幸的是，这种网络无关的方法从根本上是低效的，因为它为各种应用程序和平台提供了一种通用的功能。虽然这一指导方针在过去的30年里一直很好地服务于我们，但技术的发展和网络接口的紧张局势表明，重新审视它可能是富有成效的，拟议的研究计划的目标是探索一种替代范式，用于设计分布式系统，以利用新兴的软件定义网络（SDN）技术的能力。新的范例将共同设计分布式系统的逻辑和网络支持，以实现更高效、可扩展和可靠的系统。实现这一愿景需要我们在构建、工程、测试、保护和部署系统的方法上进行重大转变。**我计划建立一个领先的研究小组，将探索这种新的范式，以设计一个完整的现代数据中心软件/硬件堆栈，包括构建，测试和部署系统;探索新的安全，管理，会计和QoS技术;并研究新的编程框架，特定于域的语言，以及测试和调试工具。这项发现补助金的资助有助于引导这个研究小组。** 当前的研究背景是分布式存储系统的网络集成设计，重点是KV存储。为此，该研究计划旨在（1）描述当前KV系统的特征，以了解其网络负载和一致性属性;（2）设计和原型网络集成KV存储;（3）探索端到端QoS支持的设计;（4）研究偏离端到端原则对安全属性的影响。我们将探索的创新范式有可能对技术产生广泛的影响，包括许多建立KV商店的加拿大公司（例如，爱立信和IBM）以及150多个使用这些系统的加拿大数据中心。具体而言，该计划将产生四个主要贡献。首先，由此产生的设计将为未来的系统提供蓝图，并为停滞不前的QoS和可扩展性问题提供新的解决方案。其次，该计划将为核心分布式系统机制（例如，QoS、安全性、一致性和负载平衡）。第三，该计划将提供有关SDN API适用性的及时反馈，以实现更高效的设计。第四，该计划将为培训尖端网络和存储技术的高素质人才提供宝贵的机会。

The end-to-end design “principle” pervades the design of virtually every modern distributed system. In its extreme form, critical functionality is implemented solely in end hosts, with a dumb but fast network to connect them. Unfortunately, such a network-oblivious approach is fundamentally inefficient, as it provides one-size-fits-all functionality to a multifarious set of applications and platforms.******While this guideline has served us well in the past 30 years, technology evolution and tensions at the network interface suggest that it may be productive to revisit it. The goal of the proposed research program is to explore an alternative paradigm for designing distributed systems to leverage the capabilities of the emerging technology of software-defined networks (SDNs). The new paradigm will co-design distributed systems' logic and networking support to realize more efficient, scalable, and reliable systems. Achieving this vision requires a drastic shift in our approach to building, engineering, testing, securing, and deploying systems.******I plan to establish a leading research group that will explore this new paradigm for designing a complete modern data center software/hardware stack, including building, testing, and deploying systems; exploring new security, management, accounting, and QoS techniques; and investigating new programming frameworks, domain-specific languages, and testing and debugging tools. This Discovery Grant funding is instrumental in bootstrapping this research group. ******The immediate research context is a network-integrated design for distributed storage systems, focusing on KV stores. To this end, the research program aims to (1) characterize the current KV systems to understand their network load and consistency properties; (2) design and prototype a network-integrated KV store; (3) explore designs for end-to-end QoS support; and (4) study the impact of departing from the end-to-end principle on security properties.******The innovative paradigm we will explore has the potential to have a wide influence on technology, including many Canadian companies that build KV stores (e.g., NetApp, Ericsson, and IBM) and more than 150 Canadian data centers that use these systems. In particular, the program will result in four main contributions. First, the resulting designs will provide a blueprint for future systems and present novel solutions for stagnant QoS and scalability problems. Second, this program will build network-integrated designs for core distributed system mechanisms (e.g., QoS, security, consistency, and load balancing) that are applicable beyond KV stores. Third, this program will provide timely feedback on the suitability of the SDN API for enabling more efficient designs. Fourth, this program will provide an invaluable opportunity for training highly qualified personnel on cutting-edge networking and storage technology.