Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs
Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs
复制标题
DOI：
10.48550/arxiv.2204.12013
复制
发表时间：
2022-04
期刊：
影响因子：
--
通讯作者：
John Thorpe;Pengzhan Zhao;Jon Eyolfson;Yifan Qiao;Zhihao Jia;Minjia Zhang;R. Netravali;Guoqing Harry
中科院分区：
文献类型：
--
作者：
John Thorpe;Pengzhan Zhao;Jon Eyolfson;Yifan Qiao;Zhihao Jia;Minjia Zhang;R. Netravali;Guoqing Harry
许多领域的DNN模型的规模持续增长，导致有效培训需要较高的资源，组织和各种规模的研究实验室的成本也难以承受(而且往往难以承受)。本文的目的是通过有效利用可抢占实例来显著降低训练成本。可抢占实例是指在空闲时可以低得多的价格获得，但可以在优先级用户请求时被抢占的实例。然而，要做到这一点，需要新形式的弹性和效率来应对频繁抢占的可能性-这是一种与现有检查点技术所针对的正常群集设置中的偶尔故障截然不同的故障模式。我们给出了一个分布式系统，它通过在训练流水线中引入冗余计算来解决这些挑战，即一个节点不仅在自己的层上执行计算，而且在邻居的一些层上执行计算。我们的主要见解是，训练大型模型通常需要管道并行性，在这种情况下，“管道泡沫”自然存在。竹子小心翼翼地将多余的计算填充到这些气泡中，以低成本提供弹性。在各种广泛使用的DNN模型中，与使用按需实例的设置相比，竹子在训练吞吐量方面的性能是传统检查点的3.7倍，成本降低了2.4倍。
DNN models across many domains continue to grow in size, resulting in high resource requirements for effective training, and unpalatable (and often unaffordable) costs for organizations and research labs across scales. This paper aims to significantly reduce training costs with effective use of preemptible instances, i.e., those that can be obtained at a much cheaper price while idle, but may be preempted whenever requested by priority users. Doing so, however, requires new forms of resiliency and efficiency to cope with the possibility of frequent preemptions - a failure model that is drastically different from the occasional failures in normal cluster settings that existing checkpointing techniques target. We present Bamboo, a distributed system that tackles these challenges by introducing redundant computations into the training pipeline, i.e., whereby one node performs computations over not only its own layers but also over some layers in its neighbor. Our key insight is that training large models often requires pipeline parallelism where"pipeline bubbles"naturally exist. Bamboo carefully fills redundant computations into these bubbles, providing resilience at a low cost. Across a variety of widely used DNN models, Bamboo outperforms traditional checkpointing by 3.7x in training throughput, and reduces costs by 2.4x compared to a setting where on-demand instances are used.