The 16,384-node Parallelism of 3D-CNN Training on An Arm CPU based Supercomputer
The 16,384-node Parallelism of 3D-CNN Training on An Arm CPU based Supercomputer
复制标题
基于 Arm CPU 的超级计算机上 3D-CNN 训练的 16,384 节点并行性
DOI：
10.1109/hipc53243.2021.00029
复制
发表时间：
2021
期刊：
影响因子：
--
通讯作者：
Sato Kento
中科院分区：
文献类型：
--
作者：
Tabuchi Akihiro;Shirahata Koichi;Yamazaki Masafumi;Kasagi Akihiko;Honda Takumi;Kurihara Kouji;Kawakami Kentaro;Tabaru Tsuguchika;Fukumoto Naoto;Kuroda Akiyoshi;Fukai Takaaki;Sato Kento
随着深度神经网络训练的计算成本和可用数据集的不断增加，对超级计算机上快速分布式训练的需求越来越大。然而，为新的高级超级计算机移植和调优应用程序需要大量的开发工作。因此，我们提出了在新的基于Arm CPU的超级计算机Fugaku上进行3D-CNN模型训练的软件调优最佳实践。我们(i)通过aarch64的JIT翻译器调整DL中的计算，（ii）优化集体通信，如针对6D网格/环面网络拓扑的Allreduce， （iii）通过压缩数据分段和缓存数据加载器调整i /O，以及（iv）并行化数据和模型并行化训练。我们将所提出的方法应用于CosmoFlow 3D-CNN模型，使用16384个节点，包含4096个数据并行度和4个模型并行度，在30分钟内实现了训练。这是世界上所有基于cpu的MLPerf HPC v0.7系统中最快的结果。
As the computational cost and datasets available for deep neural network training continue to increase, there is a significant demand for fast distributed training on supercomputers. However, porting and tuning applications for new advanced supercomputers requires tremendous amount of development efforts. Therefore, we present software tuning best practice for a 3D-CNN model training on a new Arm CPU based supercomputer, Fugaku. We (i) tune computation in DL by a JIT translator for aarch64, (ii) optimize collective communication such as Allreduce for 6D mesh/torus network topology, (iii) tune I/O by data staging with compression and data loader with caching, and (iv) parallelize training in data and model parallelism. We apply the proposed methods to a CosmoFlow 3D-CNN model, and achieve the training in 30 minutes using 16,384 nodes consisting of 4096 data- and 4 model-parallelism. This is the fastest result of any CPU-based systems in MLPerf HPC v0.7 in the world.