Scalable Hybrid-parallelism Design for Mega-Size Deep Learning Model

超大规模深度学习模型的可扩展混合并行设计

• 批准号: 21K17751
• 负责人: Nguyen Truong
• 金额: $ 3万
• 依托单位: National Institute of Advanced Industrial Science and Technology
• 依托单位国家: 日本
• 项目类别: Grant-in-Aid for Early-Career Scientists
• 财政年份: 2021
• 资助国家: 日本
• 起止时间: 2021-04-01 至 2024-03-31
• 项目状态: 已结题
• 来源: https://kaken.nii.ac.jp/en/grant/KAKENHI-PROJECT-21K17751/
• 关键词: Deep Learning; Large Scale; Distributed Computing; Non-IID; Large-scale; Distributed computing; Hybrid parallelism

This year, we develop new methods to reduce the computing time by eliminating non-important samples during the training process (submitted to ICML2023).Through our previous work (IPDPS2022), we found that local shuffling could not achieve good accuracy in large-scale training due to non-iid data and overfitting issues. We deal with non-iid by assigning the impact factor for the models from different workers dynamically and use knowledge distillation for dealing with overfitting. The work is the Best Paper Award Finalist in CCGRID2023.We study the method to reduce the communication time by a co-design of collective communication algorithm and the intra-node network architecture (a Q1-journal JPDC is accepted) and the inter-node network architecture (poster at HPCA-Asia2023).

今年，我们开发了新的方法，通过在训练过程中消除非重要样本来减少计算时间（提交给ICML2023）。通过我们之前的工作（IPDPS2022），我们发现局部洗牌在大规模训练中由于非id数据和过拟合问题无法达到很好的准确率。我们通过动态分配不同工作者模型的影响因子来处理非id，并使用知识蒸馏来处理过拟合。该作品是CCGRID2023最佳论文奖决赛入围者。我们研究了通过共同设计集体通信算法和节点内网络架构（JPDC接受q1期刊）和节点间网络架构（HPCA-Asia2023海报）来减少通信时间的方法。

项目成果

Effective Switchless Inter-FPGA Memory Networks

有效的无开关 FPGA 间内存网络

• 发表时间: 2023
• 期刊: Journal of Parallel and Distributed Computing
• 影响因子: 3.8
• 作者: Truong Thao Nguyen;Kien Trung Pham;Hiroshi Yamaguchi;Yutaka Urino;Michihiro Koibuchi
• 通讯作者: Michihiro Koibuchi

Why Globally Re-shuffle? Revisiting Data Shuffling in Large Scale Deep Learning

• DOI: 10.1109/ipdps53621.2022.00109
• 发表时间: 2022-05
• 期刊: 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
• 作者: Thao Nguyen;François Trahay;Jens Domke;Aleksandr Drozd;Emil;Vatai;Jianwei Liao;M. Wahib

Efficient Allreduce Algorithm for Large-Scale Deep Learning on Distributed Loop Networks

用于分布式环网络大规模深度学习的高效 Allreduce 算法

• 发表时间: 2023
• 作者: Truong Thao Nguyen;Peng Chen;Yusuke Tanimura
• 通讯作者: Yusuke Tanimura

Scalable Low-Latency Inter-FPGA Networks

可扩展的低延迟 FPGA 间网络

• 发表时间: 2022
• 期刊: 36th IEEE International Parallel & Distributed Processing Symposium
• 作者: Kien Trung Pham;Truong Thao Nguyen;Hiroshi Yamaguchi;Yutaka Urino;Michihiro Koibuchi
• 通讯作者: Michihiro Koibuchi

CADIS: Handling Cluster-skewed Non-IID Data in Federated Learning with Clustered Aggregation and Knowledge DIStilled Regularization

• DOI: 10.1109/ccgrid57682.2023.00032
• 发表时间: 2023-02
• 期刊: 2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid)
• 作者: Nang Hung Nguyen;Duc Long Nguyen;Trong Bang Nguyen;T. Nguyen;H. Pham;Truong Thao Nguyen;Phi-Le Nguyen-P