Collaborative Research: OAC Core: ScaDL: New Approaches to Scaling Deep Learning for Science Applications on Supercomputers

今天的深度学习（DL）革命是由高效的深度神经网络（DNN）训练方法实现的，这些方法可以在紧凑、易于使用的DNN模型中捕获大量数据中的重要模式。深度学习方法通常应用于自然语言翻译和图像标记等任务，在科学和工程领域，还应用于药物设计、环境监测和聚变能等各种问题。然而，随着数据量的增加和深度学习方法的复杂化，训练新模型所需的时间往往成为一个主要挑战。可扩展深度学习（ScaDL）项目将通过使用专门的高性能计算（HPC）系统更快地训练更大的模型来解决这一挑战。由于通信成本随着处理器数量的增加而迅速增长，在现代高性能计算系统中有效地利用数千个强大的处理器进行深度神经网络训练一直受到阻碍。ScaDL将通过开发新的DNN训练方法来克服这一障碍，这些方法通过执行额外的计算来减少通信需求，通过在一系列以不同方式使用深度学习的科学应用中验证这些新方法的有效性，并通过将新方法集成到可扩展的深度学习软件中，供领域科学家、计算机科学家和在HPC中心支持深度学习应用的工程师使用。通过允许使用强大的HPC系统以比单台计算机快数千倍的速度训练DNN模型，ScaDL将在许多科学和工程领域取得进步。该项目还将通过吸引博士生参与项目目标，通过在芝加哥大学的新DL系统工程课程中使用ScaDL工具，以及通过在德克萨斯高级计算中心（TACC）和芝加哥大学的暑期学校招募参与者来促进教育成果，这两个学校都旨在招募来自服务不足社区的研究生，本科生和高中水平的学生，将这些工具应用于科学问题。ScaDL对科学应用和教育的关注与NSF促进科学进步的使命保持一致。ScaDL项目以两种方式为科学做出贡献。首先，它探索了在不损失模型性能的情况下提高常用优化方法的速度和可扩展性的新技术，通过：1)利用二阶信息近似的可扩展算法；2)通过调整计算和通信，开发适应不同计算机硬件的方法，使训练速度最大化；3)探索压缩技术以减少通信开销；4)使用知名的基准应用程序来评估ScaDL的收敛性；5)将其新算法和系统应用于科学应用。其次，它将发布所提出算法和系统的开源实现。该实现将可在各种硬件平台上使用，并能够选择所需的计算和通信比例，以有效地利用特定HPC系统上的计算和通信硬件。由此产生的算法和系统将有助于将ScaDL研究成果传播给广泛的研究领域和用户，并促进新方法在实际环境中的采用。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Today's deep learning (DL) revolution is enabled by efficient deep neural network (DNN) training methods that capture important patterns within large quantities of data in compact, easily usable DNN models. DL methods are applied routinely to tasks like natural language translation and image labeling--and, in science and engineering, to problems as diverse as drug design, environmental monitoring, and fusion energy. Yet as data sizes increase and DL methods grow in sophistication, the time required to train new models often emerges as a major challenge. The Scalable Deep Learning (ScaDL) project will address this challenge by making it possible to use specialized high-performance computing (HPC) systems to train bigger models more rapidly. Efficient use of the thousands of powerful processors in modern HPC systems for DNN training has previously been stymied by communication costs that grow rapidly with the number of processors used. ScaDL will overcome this obstacle by developing new DNN training methods that reduce communication requirements by performing additional computation, by validating the effectiveness of these new methods in a range of scientific applications that use DL in different ways, and by integrating the new methods into scalable DL software for use by domain scientists, computer scientists, and engineers supporting DL application in HPC centers. By permitting the use of powerful HPC systems to train DNN models thousands of times faster than on a single computer, ScaDL will enable advances in many areas of science and engineering. The project will also contribute to educational outcomes by engaging PhD students in project goals, by using ScaDL tools in a new DL systems engineering class at the University of Chicago, and by enlisting participants in summer schools at the Texas Advanced Computing Center (TACC) and U. Chicago, both of which target recruitment of students from underserved communities at the graduate, undergraduate, and high-school levels, to apply the tools to scientific problems. ScaDL's focus on science applications and education aligns the project with NSF's mission of promoting the progress of science.The ScaDL project contributes to science in two ways. First, it explores new techniques for enhancing the speed and scalability of commonly used optimization methods without losing model performance, by: 1) exploiting scalable algorithms for second-order information approximation; 2) developing methods for adapting to different computer hardware by tuning computation and communication to maximize training speed; 3) exploring compression techniques to reduce communication overheads; 4) using well-known benchmark applications to evaluate the convergence of ScaDL; and 5) applying its new algorithms and systems to science applications. Second, it will release an open-source implementation of the proposed algorithms and system. The implementation will be available on a variety of hardware platforms and capable of choosing the ratio of computation and communication required to make efficient use of the computation and communication hardware on a particular HPC system. The resulting algorithms and system will help disseminate ScaDL research results to a wide spectrum of research domains and users, and promote the adoption of the new methods in practical settings.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.