Collaborative Research: OAC Core: ScaDL: New Approaches to Scaling Deep Learning for Science Applications on Supercomputers

协作研究：OAC 核心：ScaDL：在超级计算机上扩展深度学习科学应用的新方法

• 批准号: 2401246
• 负责人: Zhao Zhang
• 金额: $ 22.64万
• 依托单位: Rutgers University New Brunswick
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2023
• 资助国家: 美国
• 起止时间: 2023-10-01 至 2024-10-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2401246&HistoricalAwards=false
• 关键词: Collaborative; Research; OAC; Core; ScaDL

Today's deep learning (DL) revolution is enabled by efficient deep neural network (DNN) training methods that capture important patterns within large quantities of data in compact, easily usable DNN models. DL methods are applied routinely to tasks like natural language translation and image labeling--and, in science and engineering, to problems as diverse as drug design, environmental monitoring, and fusion energy. Yet as data sizes increase and DL methods grow in sophistication, the time required to train new models often emerges as a major challenge. The Scalable Deep Learning (ScaDL) project will address this challenge by making it possible to use specialized high-performance computing (HPC) systems to train bigger models more rapidly. Efficient use of the thousands of powerful processors in modern HPC systems for DNN training has previously been stymied by communication costs that grow rapidly with the number of processors used. ScaDL will overcome this obstacle by developing new DNN training methods that reduce communication requirements by performing additional computation, by validating the effectiveness of these new methods in a range of scientific applications that use DL in different ways, and by integrating the new methods into scalable DL software for use by domain scientists, computer scientists, and engineers supporting DL application in HPC centers. By permitting the use of powerful HPC systems to train DNN models thousands of times faster than on a single computer, ScaDL will enable advances in many areas of science and engineering. The project will also contribute to educational outcomes by engaging PhD students in project goals, by using ScaDL tools in a new DL systems engineering class at the University of Chicago, and by enlisting participants in summer schools at the Texas Advanced Computing Center (TACC) and U. Chicago, both of which target recruitment of students from underserved communities at the graduate, undergraduate, and high-school levels, to apply the tools to scientific problems. ScaDL's focus on science applications and education aligns the project with NSF's mission of promoting the progress of science.The ScaDL project contributes to science in two ways. First, it explores new techniques for enhancing the speed and scalability of commonly used optimization methods without losing model performance, by: 1) exploiting scalable algorithms for second-order information approximation; 2) developing methods for adapting to different computer hardware by tuning computation and communication to maximize training speed; 3) exploring compression techniques to reduce communication overheads; 4) using well-known benchmark applications to evaluate the convergence of ScaDL; and 5) applying its new algorithms and systems to science applications. Second, it will release an open-source implementation of the proposed algorithms and system. The implementation will be available on a variety of hardware platforms and capable of choosing the ratio of computation and communication required to make efficient use of the computation and communication hardware on a particular HPC system. The resulting algorithms and system will help disseminate ScaDL research results to a wide spectrum of research domains and users, and promote the adoption of the new methods in practical settings.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

当今的深度学习（DL）革命是由高效的深度神经网络（DNN）训练方法实现的，这些方法可以在紧凑、易于使用的DNN模型中捕获大量数据中的重要模式。深度学习方法通常应用于自然语言翻译和图像标记等任务，在科学和工程领域，它也应用于药物设计、环境监测和聚变能等各种问题。然而，随着数据大小的增加和深度学习方法的复杂性，训练新模型所需的时间往往成为一个主要挑战。可扩展深度学习（ScaDL）项目将通过使用专门的高性能计算（HPC）系统来更快地训练更大的模型来解决这一挑战。在DNN训练中，现代HPC系统中数千个功能强大的处理器的有效使用以前一直受到通信成本的阻碍，通信成本随着所使用的处理器数量的快速增长而迅速增长。ScaDL将通过开发新的DNN训练方法来克服这一障碍，这些方法通过执行额外的计算来减少通信需求，通过验证这些新方法在以不同方式使用DL的一系列科学应用中的有效性，以及通过将新方法集成到可扩展的DL软件中，供领域科学家，计算机科学家和工程师在HPC中心支持DL应用。通过允许使用强大的HPC系统来训练DNN模型，速度比在单台计算机上快数千倍，ScaDL将使许多科学和工程领域取得进步。该项目还将通过让博士生参与项目目标、在芝加哥大学新的DL系统工程课程中使用ScaDL工具以及在德克萨斯州高级计算中心（TACC）和美国大学暑期学校招募参与者来促进教育成果。芝加哥，这两个目标的学生招聘从服务不足的社区在研究生，本科和高中水平，应用工具的科学问题。ScaDL对科学应用和教育的关注使该项目与NSF促进科学进步的使命保持一致。首先，它探索了在不损失模型性能的情况下提高常用优化方法的速度和可扩展性的新技术，通过：1）开发用于二阶信息近似的可扩展算法; 2）通过调整计算和通信以最大化训练速度来开发用于适应不同计算机硬件的方法; 3）探索压缩技术以减少通信开销; 4）使用著名的基准应用程序来评估ScaDL的收敛性; 5）将其新算法和系统应用于科学应用。 其次，它将发布所提出的算法和系统的开源实现。该实现将在各种硬件平台上可用，并且能够选择有效利用特定HPC系统上的计算和通信硬件所需的计算和通信比率。由此产生的算法和系统将有助于传播ScaDL的研究成果，以广泛的研究领域和用户，并促进采用新的方法在实际settings.This奖项反映了NSF的法定使命，并已被认为是值得通过评估使用基金会的智力价值和更广泛的影响审查标准的支持。