CNS Core: Small: Optimizing Distributed Machine Learning for Transient Resources using Loose Synchronization

许多领域中大规模数据集的可用性推动了云平台上大规模分布式机器学习（ML）工作负载的增长，以便从这些数据中获取见解。为了降低执行这些工作负载的成本，云平台已经开始以很高的折扣价格提供临时服务器。不幸的是，云平台可能随时撤销临时服务器，这可能会降低分布式机器学习性能并消除任何成本效益。高撤销率对于支持同步处理的分布式ML工作负载来说尤其有问题，因为被撤销的服务器会阻止其他服务器继续越过预定义的同步屏障，直到替换服务器能够到达该屏障。虽然异步处理消除了这种阻塞并提高了性能，但它不保持同步算法的算法属性，导致算法收敛速度较慢或可能阻止收敛。为了在低成本的瞬时服务器上保持性能，该项目建议重新设计传统的分布式ML算法，以使用更松散的同步形式。这种松散的同步通过保持同步处理的算法收敛特性来消除同步处理和异步处理之间的差距，同时使一些异步处理能够避免阻塞。该项目将这种松散的同步方法与自适应策略结合起来，根据性能、成本和波动性选择临时服务器，以显著降低在云平台上执行大规模分布式机器学习工作负载的成本。从大规模数据集中获得洞察力的分布式机器学习（ML）工作负载已成为多个行业领域众多进步的基础。该项目有可能通过显著降低成本和提高使用临时服务器在云平台上执行分布式机器学习工作负载的效率来加速这些进步。为了使更广泛的社区受益，该项目将公开发布其软件工件作为开源。该项目将把关于瞬态服务器和分布式机器学习的主题纳入分布式和操作系统的研究生和本科生课程。该项目还将通过相关的暑期研究体验项目和本科论文，让本科生参与研究。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The availability of large-scale data sets in many domains has driven the growth of large-scale distributed machine learning (ML) workloads on cloud platforms to derive insights from this data. To reduce the cost of executing these workloads, cloud platforms have begun to offer transient servers for a highly discounted price. Unfortunately, cloud platforms may revoke transient servers at any time, which can decrease distributed ML performance and eliminate any cost benefit. High revocation rates are especially problematic for distributed ML workloads that support synchronous processing, since revoked servers block others from continuing past predefined synchronization barriers until a replacement server can reach the barrier. While asynchronous processing eliminates this blocking and improves performance, it does not maintain the algorithmic properties of synchronous algorithms, resulting in slower algorithmic convergence or possibly preventing convergence. To maintain performance on low-cost transient servers, this project proposes re-designing traditional distributed ML algorithms to use looser forms of synchrony. Such loose synchronization minds the gap between synchronous and asynchronous processing by maintaining the algorithmic convergence properties of synchronous processing, while enabling some asynchronous processing to avoid blocking. The project combines this loose synchronization approach with adaptive policies for selecting transient servers based on their performance, cost, and volatility to significantly reduce the cost of executing large-scale distributed ML workloads on cloud platforms.Distributed machine learning (ML) workloads that derive insights from large-scale data sets have become the foundation for numerous advances across multiple industry sectors. This project has the potential to accelerate these advances by significantly decreasing the cost and improving the efficiency of executing distributed ML workloads on cloud platforms using transient servers. To benefit the broader community, the project will publicly release its software artifacts as open source. The project will incorporate topics on transient servers and distributed ML into graduate and undergraduate courses on distributed and operation systems. The project will also involve undergraduates in research through related summer research experience projects and undergraduate theses.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.