Collaborative Research: OAC Core: ScaDL: New Approaches to Scaling Deep Learning for Science Applications on Supercomputers

今天的深度学习(DL)革命是由高效的深度神经网络(DNN)训练方法实现的，这些方法在紧凑、易于使用的DNN模型中捕获大量数据中的重要模式。数字语言方法通常被应用于自然语言翻译和图像标记等任务，在科学和工程中，也被应用于药物设计、环境监测和聚变能源等各种各样的问题。然而，随着数据量的增加和数据挖掘方法的复杂程度的提高，训练新模型所需的时间往往成为一个重大挑战。可扩展深度学习(ScaDL)项目将通过使用专门的高性能计算(HPC)系统更快地训练更大的模型来应对这一挑战。在现代HPC系统中，有效使用数千个强大的处理器进行DNN培训之前一直受到通信成本的阻碍，通信成本随着处理器数量的增加而迅速增长。ScaDL将克服这一障碍，方法是开发新的DNN培训方法，通过执行额外的计算来减少通信需求，验证这些新方法在以不同方式使用DL的一系列科学应用中的有效性，并将新方法集成到可扩展的DL软件中，供领域科学家、计算机科学家和支持HPC中心中的DL应用的工程师使用。通过允许使用强大的HPC系统来训练DNN模型，速度比在一台计算机上快数千倍，ScaDL将使科学和工程的许多领域取得进展。该项目还将通过让博士生参与项目目标，通过在芝加哥大学一个新的数字图书馆系统工程课程中使用ScaDL工具，以及通过招募德克萨斯高级计算中心(TACC)和芝加哥大学暑期班的参与者(这两个中心的目标都是从研究生、本科生和高中水平的服务不足的社区招募学生)来促进教育成果，以将工具应用于科学问题。ScaDL对科学应用和教育的关注使该项目与NSF促进科学进步的使命保持一致。ScaDL项目在两个方面为科学做出贡献。首先，它探索了在不损失模型性能的情况下提高常用优化方法的速度和可扩展性的新技术：1)利用可扩展的二阶信息近似算法；2)通过调整计算和通信以最大化训练速度来开发适应不同计算机硬件的方法；3)探索压缩技术以减少通信开销；4)使用著名的基准应用来评估ScaDL的收敛；以及5)将其新的算法和系统应用于科学应用。其次，它将发布所提出的算法和系统的开源实现。该实施将可在各种硬件平台上使用，并能够选择有效利用特定高性能计算系统上的计算和通信硬件所需的计算和通信比率。由此产生的算法和系统将有助于将ScaDL研究成果传播到广泛的研究领域和用户，并促进在实际环境中采用新方法。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Today's deep learning (DL) revolution is enabled by efficient deep neural network (DNN) training methods that capture important patterns within large quantities of data in compact, easily usable DNN models. DL methods are applied routinely to tasks like natural language translation and image labeling--and, in science and engineering, to problems as diverse as drug design, environmental monitoring, and fusion energy. Yet as data sizes increase and DL methods grow in sophistication, the time required to train new models often emerges as a major challenge. The Scalable Deep Learning (ScaDL) project will address this challenge by making it possible to use specialized high-performance computing (HPC) systems to train bigger models more rapidly. Efficient use of the thousands of powerful processors in modern HPC systems for DNN training has previously been stymied by communication costs that grow rapidly with the number of processors used. ScaDL will overcome this obstacle by developing new DNN training methods that reduce communication requirements by performing additional computation, by validating the effectiveness of these new methods in a range of scientific applications that use DL in different ways, and by integrating the new methods into scalable DL software for use by domain scientists, computer scientists, and engineers supporting DL application in HPC centers. By permitting the use of powerful HPC systems to train DNN models thousands of times faster than on a single computer, ScaDL will enable advances in many areas of science and engineering. The project will also contribute to educational outcomes by engaging PhD students in project goals, by using ScaDL tools in a new DL systems engineering class at the University of Chicago, and by enlisting participants in summer schools at the Texas Advanced Computing Center (TACC) and U. Chicago, both of which target recruitment of students from underserved communities at the graduate, undergraduate, and high-school levels, to apply the tools to scientific problems. ScaDL's focus on science applications and education aligns the project with NSF's mission of promoting the progress of science.The ScaDL project contributes to science in two ways. First, it explores new techniques for enhancing the speed and scalability of commonly used optimization methods without losing model performance, by: 1) exploiting scalable algorithms for second-order information approximation; 2) developing methods for adapting to different computer hardware by tuning computation and communication to maximize training speed; 3) exploring compression techniques to reduce communication overheads; 4) using well-known benchmark applications to evaluate the convergence of ScaDL; and 5) applying its new algorithms and systems to science applications. Second, it will release an open-source implementation of the proposed algorithms and system. The implementation will be available on a variety of hardware platforms and capable of choosing the ratio of computation and communication required to make efficient use of the computation and communication hardware on a particular HPC system. The resulting algorithms and system will help disseminate ScaDL research results to a wide spectrum of research domains and users, and promote the adoption of the new methods in practical settings.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.