NeTS: Small: A Scalable and Efficient Architecture for Exploiting Physical Layer Optics for High Performance Multicast in Data Centers

随着数据中心规模的扩大，对快速数据复制以跨服务器分配计算负载的需求日益增长。对于分布式机器学习算法尤其如此，在分布式机器学习算法中，一台“主计算机”必须将计算更新(即，进行多播)分发给100台“S”或甚至1000台“S”的其他服务器。例如，对于文本挖掘，计算的每一次迭代都很容易导致分发超过1 GB的数据，总数超过1 TB。遗憾的是，在数据中心传输这些海量数据集的最先进解决方案依赖于应用层覆盖，在应用层覆盖中，数据以大量重复流的形式传输，这是次优的。该项目寻求实现数据中心组播性能的根本性突破，利用光传输来执行组播功能，通过可重新配置的光学设备同时将源机架连接到多个接收机架。这项工作的结果将通过更高效的数据中心运营，影响日益推动美国经济的数据分析。该项目开发了一个可扩展的架构，利用分布式小端口数光交叉连接交换机和分光器来实现高效支持单播和组播流量的数据中心网络。为了充分发挥这种体系结构的潜力，本项目设计、分析和评估了：(1)可扩展的、高性能的光分路器网络连接，解决了由于分组交换端口数、链路速度和分路器配置不同而造成的困难；(2)针对不同的网络连接和工作负载，不同的路由算法及其性能权衡，解决了故障恢复等挑战；(3)以优化应用层性能为目标的组播需求分配和光组播网络资源调度的不同算法；(4)利用分布式小端口数光电路交换机的灵活性以适应单播和组播流量热点之间的动态变化的不同方法；(5)在此新架构上运行的各种真实大数据应用程序，以解决系统设计和实施方面的挑战。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

As data centers scale up, there is an increasing need for fast data replication to distribute computational load across servers. This is especially true of distributed machine learning algorithms, where a 'master computer' must distribute computational updates (i.e., do a multicast) to 100's or even 1000's of other servers. For example, for text mining, each iteration of a computation can easily result in more than a 1 Gigabyte of data being distributed, with the total exceeding 1 Terabyte. Unfortunately, the state-of-the-art solutions for transporting these massive data sets in the data center rely on application-layer overlays where the data is transmitted sub-optimally as a large number of repetitive flows. This project seeks to realize a fundamental breakthrough in data center multicast performance by leveraging optical transmission to perform the multicast function by simultaneously connecting a source machine rack to multiple receiver machine racks with a reconfigurable optical device. The results of this work will impact data analytics, which increasingly drives the US economy, through much more efficient operation of datacenters. This project develops a scalable architecture leveraging distributed small port-count optical cross connect switches and optical splitters to realize a data center network that efficiently supports both unicast and multicast traffic. To fully achieve the potential of this architecture, this project designs, analyzes and evaluates: (1) Scalable, high performance network connectivity for an optical splitter network, addressing difficulties resulting from heterogeneous packet switch port-counts, link speeds and splitter configurations; (2) Different routing algorithms and their performance trade-offs for different network connectivity and workload, addressing challenges such as failure recovery; (3) Different algorithms for multicast demand placement and optical multicast network resource scheduling that aim to optimize application level performance; (4) Different approaches to exploit the flexibility of the distributed small port-count optical circuit switches for adapting to dynamic shifts between unicast and multicast traffic hot-spots; (5) A variety of real big data applications operating over this new architecture, addressing the system design and implementation challenges.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.