Pathways: Asynchronous Distributed Dataflow for ML
Pathways: Asynchronous Distributed Dataflow for ML
复制标题
DOI：
10.48550/arxiv.2203.12533
复制
发表时间：
2022-03
期刊：
影响因子：
--
通讯作者：
Paul Barham;Aakanksha Chowdhery;J. Dean;Sanjay Ghemawat;S. Hand;D. Hurt;M. Isard;Hyeontaek Lim;Ruoming Pang;Sudip Roy;Brennan Saeta;Parker Schuh;Ryan Sepassi;Laurent El Shafey;C. Thekkath;Yonghui Wu
中科院分区：
文献类型：
--
作者：
Paul Barham;Aakanksha Chowdhery;J. Dean;Sanjay Ghemawat;S. Hand;D. Hurt;M. Isard;Hyeontaek Lim;Ruoming Pang;Sudip Roy;Brennan Saeta;Parker Schuh;Ryan Sepassi;Laurent El Shafey;C. Thekkath;Yonghui Wu
我们提出了一种新的大规模加速器编排层的设计。我们的系统Pathways旨在探索新系统和机器学习研究思路，同时保留当前模型的最先进性能。Pathways使用异步操作符的分片数据流图来消费和产生未来，并有效地在数千个加速器上进行异构并行计算，同时在专用互连上协调数据传输。Pathways使用了一种新的异步分布式数据流设计，该设计允许控制平面并行执行，而不考虑数据平面中的依赖关系。这种设计经过精心设计，允许Pathways采用单控制器模型，从而更容易表达复杂的新并行模式。我们证明，当在2048个tpu上运行SPMD计算时，Pathways可以与最先进的系统实现性能对等（~100%的加速器利用率），同时还提供了与Transformer模型的SPMD案例相当的吞吐量，这些模型跨16级流水线，或在数据中心网络上连接的两个加速器孤岛上分片。
We present the design of a new large scale orchestration layer for accelerators. Our system, Pathways, is explicitly designed to enable exploration of new systems and ML research ideas, while retaining state of the art performance for current models. Pathways uses a sharded dataflow graph of asynchronous operators that consume and produce futures, and efficiently gang-schedules heterogeneous parallel computations on thousands of accelerators while coordinating data transfers over their dedicated interconnects. Pathways makes use of a novel asynchronous distributed dataflow design that lets the control plane execute in parallel despite dependencies in the data plane. This design, with careful engineering, allows Pathways to adopt a single-controller model that makes it easier to express complex new parallelism patterns. We demonstrate that Pathways can achieve performance parity (~100% accelerator utilization) with state-of-the-art systems when running SPMD computations over 2048 TPUs, while also delivering throughput comparable to the SPMD case for Transformer models that are pipelined across 16 stages, or sharded across two islands of accelerators connected over a data center network.