CRII: OAC: Scalability of Deep-Learning Methods on HPC Systems: An I/O-centric Approach

在过去的几年里，机器学习（ML）算法已经成为许多科学领域的关键技术。由于最近图形处理单元（GPU）的民主化，机器学习主要由深度学习（DL）技术推动，这些技术需要广泛的计算能力和大量的数据。训练大型深度神经网络（DNN）是计算密集型的。然而，由于GPU，训练的计算密集型组件的成本已经降低，而内存访问的相对成本随着输入数据集大小和ML模型复杂性的巨大增加而增加。由于对计算和内存能力的要求越来越高，DNN现在在分布式系统上进行训练，最近受到高性能计算（HPC）社区的关注。在极端规模下，HPC系统面临的一个关键挑战是通信瓶颈，因为通信比所需的计算慢得多，并且还导致了大规模机器的高能耗。缺乏对ML算法引起的不同权衡、成本和影响的全面理解，可能会在不久的将来严重损害科学发现和人工智能突破。该项目旨在通过开发准确的性能模型来解决这个问题，这些模型可以捕获在I/O（通信）方面大规模训练DNN的复杂性，并基于这些模型，在HPC机器上训练DNN时产生有效的调度算法以减少通信。在训练阶段减少数据交换可以减少这个昂贵过程的执行时间，也可能减少其能耗。DNN的培训对于许多科学领域来说越来越重要，因此优化这一关键组件的执行将有助于NSF履行其推动和促进科学进步的使命。拟议的研究将为研究人员提供性能模型，这些模型是支持HPC平台上大规模ML的新型中间件系统开发的关键。教育和外联活动将包括开发教学模块，向学生讲授分布式计算和大型神经网络培训的关键概念，并使学生能够参加为社区服务的讲习班和会议。在分布式HPC系统上训练大型神经网络具有挑战性。DNN训练涉及复杂的通信模式，由于用于解决网络的优化方法具有一定的随机性，大多数情况下是随机梯度下降（SGD）。大多数分布式机器学习被设计为在云基础设施上运行，但是HPC机器在具有快速互连网络和高级通信功能（例如远程直接内存访问（RDMA））的硬件方面表现出不同的特征，并且在使用消息传递接口（MPI）和OpenMP并行编程模型的软件方面表现出不同的特征。该项目将设计考虑HPC特征的性能模型，这些特征将对DNN大规模训练的行为提供有用的见解，例如，数据通信量如何随着DNN批量大小而变化，或者如何利用HPC多层存储（如突发缓冲区）来提高DNN训练性能。 该项目围绕三个研究重点组织：（i）在HPC机器上训练DNN时估计数据移动成本（ii）使用能量指标增强性能模型，以及（iii）开发双目标算法，最大限度地减少通信和能量，同时仍然提供训练准确性保证。为了解决这三个研究重点，本项目将采用模拟驱动的方法。第一步将是描述DNN在HPC机器上训练时的I/O行为。基于对收集到的数据的分析，设计了几种性能模型和调度算法。然后，将使用NSF资助的WRENCH项目开发HPC机器的模拟器。该模拟器将使用表征阶段收集的数据进行校准。最后，性能模型和调度算法将使用校准的模拟器进行评估。该项目还将利用模拟器不断改进性能模型和算法。该项目将为科学家提供模型，以更好地了解在复杂的分布式系统上训练大规模神经网络时产生的性能权衡。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Machine learning (ML) algorithms have became key techniques in many scientific domains over the last few years. Thanks to the recent democratization of graphics processing units (GPUs), machine learning is mostly fueled by deep learning (DL) techniques that require extensive computational capabilities and vast volumes of data. Training large deep neural networks (DNN) is compute-intensive. However, thanks to GPUs, the cost of the compute-intensive components of the training has been reduced, while the relative cost of memory accesses has increased following the huge increase in the size of inputs datasets and in the complexity of the ML models. Due to their increasing requirements in terms of computational and memory capabilities, DNNs are now trained on distributed systems and have recently gained attention from the high-performance computing (HPC) community. A key challenge on HPC systems at extreme scale is the communication bottleneck, as communication is much slower than the required computations and also accounts for high energy consumption on large-scale machines. A lack of a comprehensive understanding of the different trade-offs, costs, and impacts induced by ML algorithms may severely impair science discoveries and AI breakthroughs in the near future. This project aims to address this problem by developing accurate performance models that can capture the complexity of training a DNN at scale in terms of I/O (communication) and, based on these models, producing efficient scheduling heuristics to reduce communication when training DNNs on HPC machines. Reducing data exchanges during the training phase decreases the execution time of this costly process and is likely to also reduce its energy consumption. The training of DNNs is becoming essential for many scientific domains, so optimizing the execution of this key component will help NSF fulfill its mission to advance and promote the progress of science. The proposed research will provide researchers with performance models that are key to supporting the development of novel middleware systems for large-scale ML on HPC platforms. Educational and outreach activities will include the development of pedagogic modules that will teach students key concepts of distributed computing and training of large neural networks and enable students to participate in workshops and conferences that serve the community. Training large neural networks on distributed HPC systems is challenging. DNN training involves complex communications patterns with some randomness due to the optimization method used to solve the network, which most of the time is stochastic gradient descent (SGD). Most distributed ML has been designed to run on cloud infrastructures, however HPC machines exhibit different characteristics in terms of hardware with fast interconnect networks and advanced communications capabilities, such as remote direct memory access (RDMA) and, in terms of software with the usage of the message passing interface (MPI) and OpenMP parallel programming models. This project will design performance models taking into account HPC characteristics that will give useful insights into the behavior of DNN training at scale, for example, how the data communication volume evolves with the DNN batch size or how to leverage HPC multi-layered storage, such as burst buffers, to improve DNN training performance. This project is organized around three research thrusts: (i) estimation of data movement costs when training DNNs on HPC machines (ii) augmenting performance models with energy metrics and (iii) developing bi-objective heuristics minimizing communication and energy while still providing training accuracy guarantees. In order to address these three research thrusts, this project will adopt a simulation-driven approach. The first step will be to characterize the I/O behaviors of DNNs when trained on HPC machines. Based on the analysis of the collected data, several performance models and scheduling heuristics will be designed. Then, a simulator of the HPC machine will be developed using the NSF-funded WRENCH project. This simulator will be calibrated with the data collected during the characterization phase. Finally, the performance models and the scheduling heuristics will be evaluated using the calibrated simulator. The project will also leverage the simulator to continuously improve the performance models and heuristics. This project will provide scientists with models to better understand performance trade-offs arising when training large-scale neural networks on complex distributed systems.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.