CNS Core: Small: Mitigating Network Bottlenecks via Programmability for Distributed Machine Learning Systems

分布式机器学习(ML)正在成为一种重要的方法，它允许多个学习代理同时在同一数据集的不同切片上进行训练，并通过网络定期相互交流所学到的知识。由于网络和处理器单元之间的巨大带宽差距，网络很可能成为这些类型系统的瓶颈。为了缓解这一问题，本项目正在开发一种分布式最大似然算法和网络系统协同设计，以适应训练算法，以更好地利用网络资源。首先，提出了一种可编程通信子系统来加速训练同步。具体地说，将对网络拥塞对分布式ML模型的影响进行全面研究，以提供独特的见解。该项目还通过整合网络内控制和探索基于网络信号动态调整学习超参数的同步方案来增强现有框架。其次，提出了一种优化利用异质计算资源的调度器。为此，正在探索确定性和基于学习的调度算法，并正在开发一种框架，使操作级别的调度能够实现更细粒度的控制。这项拟议的研究调查了网络内控制以缓解网络拥塞，这仍然是高性能计算(HPC)处理器面临的最大挑战。这将显著提高现有分布式训练框架的训练效率。此外，全面和系统的研究将为算法和系统协同设计解决方案提供见解。开发的框架还将帮助学生和研究人员开展他们的大数据研究项目。将根据拟议工作的结果开发新课程，并将在暑期在高中技术夏令营开发关于网络和分布式ML的新课程和培训课程。项目中生成的源代码、原始数据和模拟结果将以标准格式存储，并将在公共领域发布。所有数据将存档在凯斯西储大学(CWRU)的部门服务器上，以提高可用性和可靠性。该奖项反映了NSF的法定使命，并已通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Distributed machine learning (ML) is becoming an important way to allow multiple learning agents to train on separate slices of the same dataset simultaneously and exchange what they have learned with each other periodically over a network. Due to the significant bandwidth gap between network and processor units, the network is likely to become the bottleneck in these types of systems. To mitigate this issue, this project is developing a distributed ML algorithm and network system co-design to adapt training algorithms to make better use of network resources. First a programmable communication subsystem is proposed to accelerate training synchronization. Specifically, a comprehensive study on the impact of network congestion over distributed ML models will be conducted to provide unique insights. The project is also enhancing existing frameworks by integrating in-network control and exploring synchronization schemes that dynamically adjust learning hyper-parameters based on network signals. Next a scheduler that optimizes the utilization of heterogeneous computing resources is proposed. To that end, both deterministic and learning-based scheduling algorithms are being explored and a framework that enables operation-level scheduling for finer-grained control is being developed. The proposed research investigates in-network control to mitigate network congestion which remains the biggest challenge for High Performance Computing (HPC) processors. It will significantly improve the training efficiency of the existing distributed training frameworks. In addition, the comprehensive and systematic studies will provide insights to the algorithm and system co-design solutions. The developed framework will also help students and researchers in their big data research projects. New courses will be developed based on the outcomes of the proposed work and new curriculum and training sessions on networking and distributed ML will be developed in High School Tech Camps during the summer. Source code, raw data, and simulation results generated in the project will be stored in standard formats and will be published in the public domain. All data will be archived on the departmental servers at Case Western Reserve University (CWRU) for increased availability and reliability.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.