Scalable Hybrid-parallelism Design for Mega-Size Deep Learning Model

点击翻译按钮获取中文摘要

This year, we develop new methods to reduce the computing time by eliminating non-important samples during the training process (submitted to ICML2023).Through our previous work (IPDPS2022), we found that local shuffling could not achieve good accuracy in large-scale training due to non-iid data and overfitting issues. We deal with non-iid by assigning the impact factor for the models from different workers dynamically and use knowledge distillation for dealing with overfitting. The work is the Best Paper Award Finalist in CCGRID2023.We study the method to reduce the communication time by a co-design of collective communication algorithm and the intra-node network architecture (a Q1-journal JPDC is accepted) and the inter-node network architecture (poster at HPCA-Asia2023).