NeTS: Small: A Scalable and Efficient Architecture for Exploiting Physical Layer Optics for High Performance Multicast in Data Centers

随着数据中心规模的扩大，越来越需要快速的数据复制来跨服务器分配计算负载。分布式机器学习算法尤其如此，其中“主计算机”必须将计算更新（即进行多播）分发到100台甚至1000台其他服务器。例如，对于文本挖掘，计算的每次迭代很容易导致分发超过1gb的数据，总数超过1tb。不幸的是，在数据中心中传输这些海量数据集的最先进的解决方案依赖于应用程序层覆盖，其中数据作为大量重复流以次优方式传输。本项目旨在通过使用可重构光器件同时将源机架连接到多个接收机架，从而利用光传输来执行组播功能，从而实现数据中心组播性能的根本性突破。这项工作的结果将影响数据分析，通过更高效的数据中心运营，数据分析日益推动美国经济的发展。该项目开发了一种可扩展的架构，利用分布式小端口数光交叉连接交换机和光分配器来实现有效支持单播和多播流量的数据中心网络。为了充分发挥该架构的潜力，本项目设计、分析和评估：(1)光分路器网络的可扩展、高性能网络连接，解决异构分组交换机端口数、链路速度和分路器配置带来的困难；(2)针对不同网络连接和工作负载的不同路由算法及其性能权衡，解决故障恢复等挑战；(3)以优化应用层性能为目标的不同组播需求放置算法和光组播网络资源调度算法；(4)利用分布式小端口数光路交换机灵活性的不同方法，以适应单播和组播业务热点之间的动态变化；(5)多种真实大数据应用在新架构上运行，解决了系统设计和实现的挑战。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

As data centers scale up, there is an increasing need for fast data replication to distribute computational load across servers. This is especially true of distributed machine learning algorithms, where a 'master computer' must distribute computational updates (i.e., do a multicast) to 100's or even 1000's of other servers. For example, for text mining, each iteration of a computation can easily result in more than a 1 Gigabyte of data being distributed, with the total exceeding 1 Terabyte. Unfortunately, the state-of-the-art solutions for transporting these massive data sets in the data center rely on application-layer overlays where the data is transmitted sub-optimally as a large number of repetitive flows. This project seeks to realize a fundamental breakthrough in data center multicast performance by leveraging optical transmission to perform the multicast function by simultaneously connecting a source machine rack to multiple receiver machine racks with a reconfigurable optical device. The results of this work will impact data analytics, which increasingly drives the US economy, through much more efficient operation of datacenters. This project develops a scalable architecture leveraging distributed small port-count optical cross connect switches and optical splitters to realize a data center network that efficiently supports both unicast and multicast traffic. To fully achieve the potential of this architecture, this project designs, analyzes and evaluates: (1) Scalable, high performance network connectivity for an optical splitter network, addressing difficulties resulting from heterogeneous packet switch port-counts, link speeds and splitter configurations; (2) Different routing algorithms and their performance trade-offs for different network connectivity and workload, addressing challenges such as failure recovery; (3) Different algorithms for multicast demand placement and optical multicast network resource scheduling that aim to optimize application level performance; (4) Different approaches to exploit the flexibility of the distributed small port-count optical circuit switches for adapting to dynamic shifts between unicast and multicast traffic hot-spots; (5) A variety of real big data applications operating over this new architecture, addressing the system design and implementation challenges.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.