CNS Core: Small: Optimizing Distributed Machine Learning for Transient Resources using Loose Synchronization

许多领域的大规模数据集的可用性推动了云平台上大规模分布式机器学习(ML)工作负载的增长，以从这些数据中获得洞察力。为了降低执行这些工作负载的成本，云平台已经开始以极低的折扣价提供临时服务器。遗憾的是，云平台可能会随时撤销临时服务器，这可能会降低分布式ML的性能，并消除任何成本效益。高吊销率对于支持同步处理的分布式ML工作负载尤其成问题，因为被吊销的服务器会阻止其他服务器继续过去的预定义同步障碍，直到替代服务器可以达到该障碍。虽然异步处理消除了这种阻塞并提高了性能，但它不保持同步算法的算法属性，从而导致算法收敛较慢或可能阻止收敛。为了在低成本的临时服务器上保持性能，该项目建议重新设计传统的分布式ML算法，以使用更松散的同步形式。这种松散的同步通过保持同步处理的算法收敛特性，同时允许某些异步处理来避免阻塞，从而注意到同步和异步处理之间的差距。该项目将这种松散同步方法与根据性能、成本和易失性选择瞬时服务器的自适应策略相结合，显著降低了在云平台上执行大规模分布式ML工作负载的成本。从大规模数据集获得洞察的分布式机器学习(ML)工作负载已成为跨多个行业的众多进步的基础。该项目有可能通过显著降低成本并提高在使用临时服务器的云平台上执行分布式ML工作负载的效率来加速这些进步。为了让更广泛的社区受益，该项目将以开源的形式公开发布其软件构件。该项目将把有关瞬时服务器和分布式ML的主题纳入关于分布式和操作系统的研究生和本科生课程。该项目还将通过相关的暑期研究体验项目和本科生论文让本科生参与研究。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The availability of large-scale data sets in many domains has driven the growth of large-scale distributed machine learning (ML) workloads on cloud platforms to derive insights from this data. To reduce the cost of executing these workloads, cloud platforms have begun to offer transient servers for a highly discounted price. Unfortunately, cloud platforms may revoke transient servers at any time, which can decrease distributed ML performance and eliminate any cost benefit. High revocation rates are especially problematic for distributed ML workloads that support synchronous processing, since revoked servers block others from continuing past predefined synchronization barriers until a replacement server can reach the barrier. While asynchronous processing eliminates this blocking and improves performance, it does not maintain the algorithmic properties of synchronous algorithms, resulting in slower algorithmic convergence or possibly preventing convergence. To maintain performance on low-cost transient servers, this project proposes re-designing traditional distributed ML algorithms to use looser forms of synchrony. Such loose synchronization minds the gap between synchronous and asynchronous processing by maintaining the algorithmic convergence properties of synchronous processing, while enabling some asynchronous processing to avoid blocking. The project combines this loose synchronization approach with adaptive policies for selecting transient servers based on their performance, cost, and volatility to significantly reduce the cost of executing large-scale distributed ML workloads on cloud platforms.Distributed machine learning (ML) workloads that derive insights from large-scale data sets have become the foundation for numerous advances across multiple industry sectors. This project has the potential to accelerate these advances by significantly decreasing the cost and improving the efficiency of executing distributed ML workloads on cloud platforms using transient servers. To benefit the broader community, the project will publicly release its software artifacts as open source. The project will incorporate topics on transient servers and distributed ML into graduate and undergraduate courses on distributed and operation systems. The project will also involve undergraduates in research through related summer research experience projects and undergraduate theses.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.