CAREER: Efficient and Scalable Large Foundational Model Training on Supercomputers for Science

深度学习（DL）方法，特别是大型基础模型，为许多科学和工程学科（如基因组学、生物信息学、气象学和自然语言处理）中的问题提供了令人兴奋的新方法。在极端规模上训练基础模型非常耗时，容易在有限的可伸缩性下实现低利用率，并且需要人力。这个NSF CAREER项目通过创新的算法、系统和接口设计，解决了大型基础模型预训练在超级计算机上的收敛、性能和可扩展性差距。除了算法和计算机系统创新之外，该项目还通过降低大规模基础模型训练的障碍和科学深度学习的时间消耗，为转化计算机科学做出了贡献，从而使更多的科学研究能够拥抱大型基础模型。研究结果将作为开源软件公开提供给更广泛的社区，并提供有关设计和使用的全面文档，以帮助所有领域的用户。从技术上讲，这个NSF CAREER项目有四个研究和教育重点：第一个重点是新的优化技术，如一阶、二阶和混合阶优化器，以及潜在的近似技术，以提高收敛时间。第二个目标是通过设计利用梯度的时空模式的新型稀疏化算法来提高缩放效率。第三个要点考虑了一种新的复杂并行抽象，这种抽象可以跨处理器透明地部署大型模型，并且在超级计算机的当前计算、互连和I/O能力下具有近乎最佳的性能。第四个要点设计教育活动，包括分布式深度学习系统课程、深度学习教程和面向不同专业水平的学生和从业者的深度学习训练营。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning (DL) methods, especially the large foundational models, enable exciting new approaches to problems in many science and engineering disciplines, such as genomics, bioinformatics, meteorology, and natural language processing. Training foundational models at extreme scales is time-consuming, prone to low utilization with limited scalability, and human-effort demanding. This NSF CAREER project addresses the convergence, performance, and scalability gaps of large foundational model pre-training on supercomputers with innovative algorithms, systems, and interface design. In addition to the algorithm and computer system innovation, this project contributes to translational computer science by lowering the barrier of sizeable foundational model training and the time consumption of scientific deep learning, thus enabling significantly more scientific research to embrace large foundational models. The research results will be publicly available as open-source software to the broader community, with comprehensive documentation on the design and usage to help users from all domains.Technically, this NSF CAREER project has four research and educational thrusts: The first thrust focuses on new optimization techniques such as first-, second-, and mixed-order optimizers with potential approximation techniques to enhance time-to-convergence. The second thrust aims to enhance the scaling efficiency by designing novel sparsification algorithms that leverage the spatial and temporal patterns of gradients. The third thrust considers a new complex parallelism abstraction that transparently deploys large models across processors with near-optimal performance given the present capability of compute, interconnect, and I/O on a supercomputer. The fourth thrust designs educational activities, including a distributed DL system course, a DL tutorial, and a DL bootcamp targeting students and practitioners with different levels of expertise.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.