Efficient Process Arrival Pattern Aware Collective Communication for Deep Learning
Efficient Process Arrival Pattern Aware Collective Communication for Deep Learning
复制标题
用于深度学习的高效流程到达模式感知集体通信
DOI：
--
复制
发表时间：
2022
期刊：
影响因子：
--
通讯作者：
A. Afsahi
中科院分区：
文献类型：
--
作者：
Pedram Alizadeh;A. Sojoodi;Y. H. Temuçin;A. Afsahi
MPI集体通信操作在并行应用中得到了广泛的应用。因此，研究人员一直在研究如何提高其性能和可扩展性，以直接影响应用程序的性能。不幸的是，大多数这些研究都是基于这样一个前提，即所有的过程都同时达到了集体的要求。然而，一些研究表明，不平衡的进程到达模式（PAP）是普遍存在于真实的环境中，显着影响集体的表现。因此，设计PAP感知的集体算法，可以提高性能，同时具有挑战性，是非常可取的。本文沿着这些路线，但在深度学习（DL）工作负载已成为主流的背景下。本文简要描述了Horovod分布式深度学习框架中的集体通信，特别是MPI_Allreduce，并表明MPI进程的到达模式确实不平衡。然后，它提出了一个节点内的共享内存PAP感知MPI_Allreduce算法的中小型消息，其中的领导者进程是动态选择的基础上，在每次调用的集体调用的进程的到达时间。然后，我们提出了一个节点内PAP感知算法的大消息，动态构建减少时间表在每个MPI_Allreduce调用。最后，我们提出了一个PAP感知集群范围内的分层算法，这是通过利用我们的节点内PAP感知的设计，这使得它的分层性质相比，平面算法的进程之间的数据依赖性更少。所提出的算法在微基准测试和Horovod与TensorFlow应用程序上分别比原生算法提高了58%和17%。
MPI collective communication operations are used extensively in parallel applications. As such, researchers have been investigating how to improve their performance and scalability to directly impact application performance. Unfortunately, most of these studies are based on the premise that all processes arrive at the collective call simultaneously. A few studies though have shown that imbalanced Process Arrival Pattern (PAP) is ubiquitous in real environments, significantly affecting the collective performance. Therefore, devising PAP-aware collective algorithms that could improve performance, while challenging, is highly desirable. This paper is along those lines but in the context of Deep Learning (DL) workloads that have become maintstream. This paper presents a brief characterization of collective communications, in particular MPI_Allreduce, in the Horovod distributed Deep Learning framework and shows that the arrival pattern of MPI processes is indeed imbalanced. It then proposes an intra-node shared-memory PAP-aware MPI_Allreduce algorithm for small to medium messages, where the leader process is dynamically chosen based on the arrival time of the processes at each invocation of the collective call. We then propose an intra-node PAP-aware algorithm for large messages that dynamically constructs the reduction schedule at each MPI_Allreduce invocation. Finally, we propose a PAP-aware cluster-wide hierarchical algorithm, which is extended by utilizing our intra-node PAP-aware designs, that imposes less data dependency among processes given its hierarchical nature compared to flat algorithms. The proposed algorithms deliver up to 58% and 17% improvement at the micro-benchmark and Horovod with TensorFlow application over the native algorithms, respectively.
DOI：
10.1145/3392717.3392771
发表时间：
2020-06
期刊：
Proceedings of the 34th ACM International Conference on Supercomputing
影响因子：
--
作者：
通讯作者：
--
使用 SHARP 的可扩展 MPI 集合：TACC Frontera 系统的大规模性能评估
DOI：
10.1109/exampi52011.2020.00007
发表时间：
2020
期刊：
ExaMPI2020 - Workshop on Exascale MPI 2020
影响因子：
--
作者：
Ramesh, Bharath;Suresh, Kaushik Kandadi;Sarkauskas, Nick;Bayatpour, Mohammadreza;Hashmi, Jahanzeb Maqbool;Subramoni, Hari;Panda, Dhabaleswar K.
通讯作者：
Panda, Dhabaleswar K.