SPX: Collaborative Research: Scaling the Software-Defined Data Center with Network-Storage Stack Co-Design

由于软件定义和可编程，今天的计算机数据中心正在经历一种颠覆性的趋势。软件定义网络（SDN）和软件定义存储（SDS）已广泛部署在当今的数据中心中，以提供多租户网络和存储服务。然而，由于网络和存储软件之间缺乏协调，数据中心在性能、可扩展性和可靠性方面仍然存在挑战。随着数据中心随着软件定义功能的发展（迄今为止通过硬件设备完成），现在是从根本上解决这些经典挑战的合适时机。本项目采用跨层研究方法，对软件定义网络和存储进行基础系统和架构研究。所提出的研究活动将有利于大型数据中心的基础设施和平台开发。该项目将涉及积极的工业合作，为技术过渡到实践开辟一条道路。此外，该项目将把研究成果整合到以网络和存储技术为中心的新课程中。该项目还将通过各种研究项目和夏令营招募和培训代表性不足的高中生。该项目研究了一种新的软件定义架构，该架构在细化每个组件的功能的同时，共同设计SDN和SDS的系统堆栈。如果成功，该项目将从根本上推进软件定义数据中心的设计原则和技术。首先，本项目提出在SDN和SDS中细化控制平面和数据平面的功能。这样的新架构使SDN和SDF之间的状态共享成为可能，这有利于全局资源管理和I/O调度，以应对性能挑战。其次，为了克服可扩展性的挑战，本项目重新思考了存储栈并发控制的设计和实现，并提出了一种网内锁管理器来扩展并发存储访问。该项目进一步利用多核CPU芯片中的硬件并行性来加速网络数据包处理。第三，分析了现有数据复制协议的局限性，提出了一种网络内冲突检测方案，以实现复制存储的线性可扩展性和强数据一致性。最后，该项目将提出的技术应用于代表性的多租户存储服务，如分布式数据库和键值存储，以及新的分布式应用程序。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Computer data centers today are experiencing a disruptive trend by virtue of becoming software-defined and programmable. Software-defined networking (SDN) and software-defined storage (SDS) have been widely deployed in today's data centers to provide multi-tenant network and storage services. However, due to the lack of coordination between the network and storage software, there are still performance, scalability, and reliability challenges in data centers. As the data center is evolving with software-defined functions (hitherto done through hardware devices), it is the right time to fundamentally address these classical challenges. This project leverages a cross-layer research approach to conduct the fundamental systems and architecture study on the software-defined network and storage. The proposed research activities will benefit the infrastructure and platform development for large-scale data centers. The project will involve active industrial collaborations to carve a path for technology transition to practice. Furthermore, the project will integrate the research developments into new courses that center around network and storage techniques. This project will also recruit and train underrepresented and high school students through various research programs and summer camps.This project investigates a new software-defined architecture that co-designs the system stacks of both SDN and SDS while refining the functionalities of each component. If successful, the project will fundamentally advance design principles and techniques of software-defined data centers. First, the project proposes to refine the functionalities of the control and data plane in SDN and SDS. Such a new architecture enables state sharing between SDN and SDF, which facilitates the global resource management and I/O scheduling to address the performance challenge. Second, to overcome scalability challenges, this project rethinks the design and implementation of concurrency control in the storage stack and proposes an in-network lock manager to scale the concurrent storage accesses. The project further exploits the hardware parallelism available in multi-core CPU chips to speed up the network packet processing. Third, the project analyzes the limitation of existing data replication protocols and proposes an in-network conflict detection scheme to achieve both linear scalability and strong data consistency for replicated storage. Finally, the project applies the proposed techniques to representative multi-tenant storage services such as distributed databases and key-value stores, and new distributed applications.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.