NeTS: Small: A Scalable and Efficient Architecture for Exploiting Physical Layer Optics for High Performance Multicast in Data Centers

随着数据中心规模的扩大，越来越需要快速数据复制来跨服务器分配计算负载。 对于分布式机器学习算法尤其如此，其中“主计算机”必须分发计算更新（即，进行多播）到100个甚至1000个其他服务器。例如，对于文本挖掘，计算的每次迭代都可以很容易地导致超过1 TB的数据被分发，总数超过1 TB。不幸的是，用于在数据中心中传输这些海量数据集的最先进的解决方案依赖于应用层覆盖，其中数据作为大量重复流以次优方式传输。该项目旨在通过利用可重构的光器件同时将源机架连接到多个接收机机架，利用光传输来执行多播功能，从而实现数据中心多播性能的根本突破。这项工作的结果将影响数据分析，通过更有效地运营数据中心，数据分析越来越多地推动美国经济。 该项目开发了一个可扩展的架构，利用分布式小端口数光交叉连接交换机和光分路器，实现一个数据中心网络，有效地支持单播和组播流量。为了充分发挥这种架构的潜力，本项目设计、分析和评估了：（1）可扩展的、高性能的光分路器网络连接，解决了异构分组交换机端口数、链路速度和分路器配置带来的困难;（2）针对不同网络连接性和工作负载的不同路由算法及其性能权衡，解决诸如故障恢复的挑战;（3）用于组播需求放置和光组播网络资源调度的不同算法，其旨在优化应用级性能;（4）利用分布式小端口的灵活性的不同方法-计数光路交换机，用于适应单播和多播业务热点之间的动态移位;（5）在这种新架构上运行的各种真实的大数据应用，该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

As data centers scale up, there is an increasing need for fast data replication to distribute computational load across servers. This is especially true of distributed machine learning algorithms, where a 'master computer' must distribute computational updates (i.e., do a multicast) to 100's or even 1000's of other servers. For example, for text mining, each iteration of a computation can easily result in more than a 1 Gigabyte of data being distributed, with the total exceeding 1 Terabyte. Unfortunately, the state-of-the-art solutions for transporting these massive data sets in the data center rely on application-layer overlays where the data is transmitted sub-optimally as a large number of repetitive flows. This project seeks to realize a fundamental breakthrough in data center multicast performance by leveraging optical transmission to perform the multicast function by simultaneously connecting a source machine rack to multiple receiver machine racks with a reconfigurable optical device. The results of this work will impact data analytics, which increasingly drives the US economy, through much more efficient operation of datacenters. This project develops a scalable architecture leveraging distributed small port-count optical cross connect switches and optical splitters to realize a data center network that efficiently supports both unicast and multicast traffic. To fully achieve the potential of this architecture, this project designs, analyzes and evaluates: (1) Scalable, high performance network connectivity for an optical splitter network, addressing difficulties resulting from heterogeneous packet switch port-counts, link speeds and splitter configurations; (2) Different routing algorithms and their performance trade-offs for different network connectivity and workload, addressing challenges such as failure recovery; (3) Different algorithms for multicast demand placement and optical multicast network resource scheduling that aim to optimize application level performance; (4) Different approaches to exploit the flexibility of the distributed small port-count optical circuit switches for adapting to dynamic shifts between unicast and multicast traffic hot-spots; (5) A variety of real big data applications operating over this new architecture, addressing the system design and implementation challenges.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.