CAREER: Efficient and Scalable Large Foundational Model Training on Supercomputers for Science

深度学习(DL)方法，特别是大型基础模型，使许多科学和工程学科，如基因组学、生物信息学、气象学和自然语言处理的问题能够采用令人兴奋的新方法。在极端规模上培训基础模型非常耗时、利用率低、可伸缩性有限，并且需要人力。该NSF职业项目通过创新的算法、系统和界面设计，解决了在超级计算机上进行大型基础模型预培训的收敛、性能和可扩展性差距。除了算法和计算机系统创新外，该项目还通过降低大规模基础模型培训的障碍和科学深度学习的时间消耗，为翻译计算机科学做出贡献，从而使更多的科学研究能够拥抱大型基础模型。研究成果将以开源软件的形式向更广泛的社区公开，并提供有关设计和使用的全面文档，以帮助来自所有领域的用户。从技术上讲，这个NSF职业项目有四个研究和教育推动力：第一个推力专注于新的优化技术，如一阶、二阶和混合阶优化器，具有潜在的近似技术，以提高收敛时间。第二个推力旨在通过设计新的稀疏算法来提高缩放效率，该算法利用了梯度的空间和时间模式。第三个推力考虑了一种新的复杂并行抽象，它透明地跨处理器部署大型模型，在给定超级计算机上现有的计算、互连和I/O能力的情况下，具有接近最佳的性能。第四个重点是设计教育活动，包括分布式数字图书馆系统课程、数字图书馆教程和面向具有不同专业水平的学生和实践者的数字图书馆训练营。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning (DL) methods, especially the large foundational models, enable exciting new approaches to problems in many science and engineering disciplines, such as genomics, bioinformatics, meteorology, and natural language processing. Training foundational models at extreme scales is time-consuming, prone to low utilization with limited scalability, and human-effort demanding. This NSF CAREER project addresses the convergence, performance, and scalability gaps of large foundational model pre-training on supercomputers with innovative algorithms, systems, and interface design. In addition to the algorithm and computer system innovation, this project contributes to translational computer science by lowering the barrier of sizeable foundational model training and the time consumption of scientific deep learning, thus enabling significantly more scientific research to embrace large foundational models. The research results will be publicly available as open-source software to the broader community, with comprehensive documentation on the design and usage to help users from all domains.Technically, this NSF CAREER project has four research and educational thrusts: The first thrust focuses on new optimization techniques such as first-, second-, and mixed-order optimizers with potential approximation techniques to enhance time-to-convergence. The second thrust aims to enhance the scaling efficiency by designing novel sparsification algorithms that leverage the spatial and temporal patterns of gradients. The third thrust considers a new complex parallelism abstraction that transparently deploys large models across processors with near-optimal performance given the present capability of compute, interconnect, and I/O on a supercomputer. The fourth thrust designs educational activities, including a distributed DL system course, a DL tutorial, and a DL bootcamp targeting students and practitioners with different levels of expertise.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.