CAREER: Efficient and Scalable Large Foundational Model Training on Supercomputers for Science

职业：科学超级计算机上高效且可扩展的大型基础模型训练

• 批准号: 2340011
• 负责人: Zhao Zhang
• 金额: $ 59.97万
• 依托单位: Rutgers University New Brunswick
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-07-01 至 2029-06-30
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2340011&HistoricalAwards=false
• 关键词: CAREER; Efficient; Scalable; Large; Foundational

Deep learning (DL) methods, especially the large foundational models, enable exciting new approaches to problems in many science and engineering disciplines, such as genomics, bioinformatics, meteorology, and natural language processing. Training foundational models at extreme scales is time-consuming, prone to low utilization with limited scalability, and human-effort demanding. This NSF CAREER project addresses the convergence, performance, and scalability gaps of large foundational model pre-training on supercomputers with innovative algorithms, systems, and interface design. In addition to the algorithm and computer system innovation, this project contributes to translational computer science by lowering the barrier of sizeable foundational model training and the time consumption of scientific deep learning, thus enabling significantly more scientific research to embrace large foundational models. The research results will be publicly available as open-source software to the broader community, with comprehensive documentation on the design and usage to help users from all domains.Technically, this NSF CAREER project has four research and educational thrusts: The first thrust focuses on new optimization techniques such as first-, second-, and mixed-order optimizers with potential approximation techniques to enhance time-to-convergence. The second thrust aims to enhance the scaling efficiency by designing novel sparsification algorithms that leverage the spatial and temporal patterns of gradients. The third thrust considers a new complex parallelism abstraction that transparently deploys large models across processors with near-optimal performance given the present capability of compute, interconnect, and I/O on a supercomputer. The fourth thrust designs educational activities, including a distributed DL system course, a DL tutorial, and a DL bootcamp targeting students and practitioners with different levels of expertise.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习（DL）方法，特别是大型基础模型，为许多科学和工程学科（如基因组学、生物信息学、气象学和自然语言处理）中的问题提供了令人兴奋的新方法。在极端规模下训练基础模型是耗时的，易于使用率低，可扩展性有限，并且需要人工努力。这个NSF CAREER项目通过创新的算法、系统和接口设计，解决了在超级计算机上进行大型基础模型预训练的收敛性、性能和可扩展性差距。除了算法和计算机系统创新之外，该项目还通过降低大规模基础模型训练的障碍和科学深度学习的时间消耗，为转化计算机科学做出了贡献，从而使更多的科学研究能够采用大型基础模型。研究结果将作为开源软件向更广泛的社区公开，并提供有关设计和使用的全面文档，以帮助来自所有领域的用户。从技术上讲，这个NSF CAREER项目有四个研究和教育重点：第一个重点是新的优化技术，如一阶、二阶和混合阶优化器，以及潜在的近似技术，以提高收敛时间。第二个推力的目的是提高缩放效率，设计新的稀疏算法，利用梯度的空间和时间模式。第三个推力考虑了一个新的复杂的并行抽象，透明地部署大型模型跨处理器与接近最佳的性能给定的计算，互连和I/O的超级计算机上的当前能力。第四个重点是设计教育活动，包括分布式DL系统课程、DL教程和针对不同专业水平的学生和从业者的DL训练营。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。