Collaborative Research: OAC Core: CropDL - Scheduling and Checkpoint/Restart Support for Deep Learning Applications on HPC Clusters
合作研究:OAC 核心:CropDL - HPC 集群上深度学习应用的调度和检查点/重启支持
基本信息
- 批准号:2403089
- 负责人:
- 金额:$ 22.5万
- 依托单位:
- 依托单位国家:美国
- 项目类别:Standard Grant
- 财政年份:2024
- 资助国家:美国
- 起止时间:2024-10-01 至 2027-09-30
- 项目状态:未结题
- 来源:
- 关键词:
项目摘要
Machine Learning (ML) and Deep Learning (DL) (more specifically, Deep Neural Network (DNN)) workloads are beginning to dominate the High-Performance Computing (HPC) arena. Today, massive computational resources are required to train even a single state-of-the-art deep learning model (e.g., large language models or LLMs). As the need for training massive DNN models continues and expands from the private sector to NSF-supported scientists and engineers (who are more likely to use shared computing resources), efficient checkpointing is emerging as a critical need. Checkpointing not only helps deal with failures but also provides more scheduling flexibility on shared HPC resources, as a very long-running job can be broken into several shorter ones. The premise of the CropDL project is that efficient and automated application-level checkpoint and restart will be critical to facilitating the use of shared HPC clusters for long-running ML training tasks, drastically increasing the number of researchers that can successfully train large ML models for various applications. This project also contributes to education and diversity in multiple aspects, for example, 1) introducing courses (or course material) to bring attention to ML-related workloads in computer systems undergraduate and graduate education; 2) integrating research tasks from this project with synergistic research programs at universities to increase the participation of women and underrepresented minority groups; and 3) supporting and training PhD students in their research, creating momentum on systems and cyberinfrastructure research related to emerging ML workloads and popularizing integrative research that combines the properties of these workloads with the complexities of modern HPC hardware.The overarching goal of CropDL is to support application-level checkpoints/restarts of deep learning applications for better resiliency, faster average completion time, and higher resource utilization. Particularly, several properties of DL workloads (as compared to scientific computations) create distinct sets of opportunities and challenges for checkpointing: 1) limited communication patterns during parallel execution, which can enable efficient coordinated checkpoints, 2) many unique opportunities for compression of checkpoints, and possibly taking uncoordinated checkpoints, and 3) malleable execution, where restarting from a different number of nodes is possible. Based on this observation, the first direction of this project is to exploit the properties of the DNN model(s) to be trained during checkpointing. This includes asynchronous versioned checkpointing for DL applications under a wide variety of parallelism models as well as content-based data reduction (compression and sparsification) techniques to reduce checkpoint volumes. The second direction of research focuses on using current and upcoming HPC systems' resources efficiently while checkpointing. It formulates tasks, data, and I/O requirements from DL applications into DAG representations and develops methods to schedule them. It also supports efficient I/O for deep learning applications with emerging I/O platforms. The last direction is to automate checkpointing through a compilation system based on the computational graph of DL workloads. All these efforts consider a variety of parallelization schemes for DNNs, i.e., data, model, and/or pipelined parallelism.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.
机器学习(ML)和深度学习(DL)(更具体地说,深度神经网络(DNN))工作负载开始主导高性能计算(HPC)领域。今天,即使是训练一个最先进的深度学习模型(例如,大型语言模型或法学硕士)也需要大量的计算资源。随着训练大规模深度神经网络模型的需求不断扩大,并从私营部门扩展到nsf支持的科学家和工程师(他们更有可能使用共享的计算资源),高效的检查点正在成为一种关键需求。检查点不仅有助于处理故障,而且还为共享HPC资源提供了更多的调度灵活性,因为一个非常长时间运行的作业可以分成几个较短的作业。CropDL项目的前提是,高效和自动化的应用程序级检查点和重启对于促进共享HPC集群用于长期运行的机器学习训练任务至关重要,这将大大增加能够成功训练大型机器学习模型的研究人员的数量。该项目还在多个方面促进了教育和多样性,例如:1)引入课程(或课程材料),以引起人们对计算机系统本科和研究生教育中与机器学习相关的工作量的关注;2)将该项目的研究任务与大学的协同研究项目相结合,以增加妇女和未被充分代表的少数群体的参与;3)支持和培训博士生的研究,为与新兴机器学习工作负载相关的系统和网络基础设施研究创造动力,并推广将这些工作负载的特性与现代高性能计算硬件的复杂性相结合的综合研究。CropDL的总体目标是支持深度学习应用程序的应用程序级检查点/重启,以获得更好的弹性、更快的平均完成时间和更高的资源利用率。特别是,深度学习工作负载的几个属性(与科学计算相比)为检查点创造了不同的机会和挑战:1)并行执行期间有限的通信模式,这可以实现高效的协调检查点;2)压缩检查点的许多独特机会,并可能采用不协调的检查点;3)可伸缩性执行,其中可以从不同数量的节点重新启动。基于这一观察,该项目的第一个方向是利用在检查点期间训练的DNN模型的属性。这包括用于各种并行模型下的深度学习应用程序的异步版本化检查点,以及用于减少检查点数量的基于内容的数据缩减(压缩和稀疏化)技术。第二个研究方向侧重于在检查点时有效地利用当前和即将到来的HPC系统资源。它将DL应用程序中的任务、数据和I/O需求公式化为DAG表示,并开发了调度它们的方法。它还支持基于新兴I/O平台的深度学习应用程序的高效I/O。最后一个方向是通过基于DL工作负载计算图的编译系统实现自动检查点。所有这些努力都考虑了dnn的各种并行化方案,即数据、模型和/或流水线并行。该奖项反映了美国国家科学基金会的法定使命,并通过使用基金会的知识价值和更广泛的影响审查标准进行评估,被认为值得支持。
项目成果
期刊论文数量(0)
专著数量(0)
科研奖励数量(0)
会议论文数量(0)
专利数量(0)
数据更新时间:{{ journalArticles.updateTime }}
{{
item.title }}
{{ item.translation_title }}
- DOI:
{{ item.doi }} - 发表时间:
{{ item.publish_year }} - 期刊:
- 影响因子:{{ item.factor }}
- 作者:
{{ item.authors }} - 通讯作者:
{{ item.author }}
数据更新时间:{{ journalArticles.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ monograph.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ sciAawards.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ conferencePapers.updateTime }}
{{ item.title }}
- 作者:
{{ item.author }}
数据更新时间:{{ patent.updateTime }}
Weikuan Yu其他文献
Performance Evaluation of FPGA-Based Biological Applications
基于 FPGA 的生物应用的性能评估
- DOI:
- 发表时间:
2007 - 期刊:
- 影响因子:0
- 作者:
O. Storaasli;Weikuan Yu;D. Strenski;James Maltby - 通讯作者:
James Maltby
Ad Hoc File Systems for High-Performance Computing
用于高性能计算的临时文件系统
- DOI:
10.1007/s11390-020-9801-1 - 发表时间:
2020 - 期刊:
- 影响因子:1.9
- 作者:
A. Brinkmann;K. Mohror;Weikuan Yu;P. Carns;Toni Cortes;S. Klasky;Alberto Miranda;F. Pfreundt;R. Ross;Marc - 通讯作者:
Marc
JVM-Bypass for Efficient Hadoop Shuffling
用于高效 Hadoop Shuffle 的 JVM 旁路
- DOI:
10.1109/ipdps.2013.13 - 发表时间:
2013 - 期刊:
- 影响因子:0
- 作者:
Yandong Wang;Cong Xu;Xiaobing Li;Weikuan Yu - 通讯作者:
Weikuan Yu
Performance evaluation and tuning of BioPig for genomic analysis
BioPig 用于基因组分析的性能评估和调整
- DOI:
10.1145/2831244.2831252 - 发表时间:
2015 - 期刊:
- 影响因子:0
- 作者:
Lizhen Shi;Zhong Wang;Weikuan Yu;Xiandong Meng - 通讯作者:
Xiandong Meng
Understanding I/O Behavior in Scientific Workflows on High Performance Computing Systems
了解高性能计算系统上科学工作流程中的 I/O 行为
- DOI:
- 发表时间:
2019 - 期刊:
- 影响因子:0
- 作者:
Fahim Chowdhury;Francesco Di;A. Moody;Elsa Gonsiorowski;K. Mohror;Weikuan Yu - 通讯作者:
Weikuan Yu
Weikuan Yu的其他文献
{{
item.title }}
{{ item.translation_title }}
- DOI:
{{ item.doi }} - 发表时间:
{{ item.publish_year }} - 期刊:
- 影响因子:{{ item.factor }}
- 作者:
{{ item.authors }} - 通讯作者:
{{ item.author }}
{{ truncateString('Weikuan Yu', 18)}}的其他基金
SaTC: CORE: Small: Realizing Enhanced Authentication in the Mobile Era
SaTC:核心:小:实现移动时代的增强认证
- 批准号:
2131143 - 财政年份:2021
- 资助金额:
$ 22.5万 - 项目类别:
Standard Grant
IRES Track-1: I/O Research for Data-Intensive Analytics and Deep Learning
IRES Track-1:数据密集型分析和深度学习的 I/O 研究
- 批准号:
1952302 - 财政年份:2020
- 资助金额:
$ 22.5万 - 项目类别:
Standard Grant
SHF: Medium: Collaborative Research: ECC: Ephemeral Coherence Cohort for I/O Containerization and Disaggregation
SHF:媒介:协作研究:ECC:I/O 容器化和分解的临时一致性队列
- 批准号:
1763547 - 财政年份:2018
- 资助金额:
$ 22.5万 - 项目类别:
Continuing Grant
CRI: II-New: A Software Defined Infrastructure for Cross-Layer Research on Reconfigurable Architecture and Systems
CRI:II-New:用于可重构架构和系统跨层研究的软件定义基础设施
- 批准号:
1822737 - 财政年份:2018
- 资助金额:
$ 22.5万 - 项目类别:
Standard Grant
Eager: Collaborative Research: DiRecMR: Reconciling the Dichotomy of MapReduce for Efficient Speculation and Resilience
Eager:协作研究:DiRecMR:调和 MapReduce 的二分法以实现高效推测和弹性
- 批准号:
1744336 - 财政年份:2017
- 资助金额:
$ 22.5万 - 项目类别:
Standard Grant
CSR: Small: XooMR: Cross-Layer and Cross-Phase Cooperation for Fair and Efficient MapReduce
CSR:小:XooMR:跨层跨阶段合作实现公平高效的 MapReduce
- 批准号:
1564647 - 财政年份:2015
- 资助金额:
$ 22.5万 - 项目类别:
Standard Grant
EAGER: Tadoop: A Dual-Purpose Framework Taming the Bipolarity of Storage and Communication for High-Performance Computing and Data Analytics
EAGER:Tadoop:一个双用途框架,克服存储和通信的两极性,实现高性能计算和数据分析
- 批准号:
1561041 - 财政年份:2015
- 资助金额:
$ 22.5万 - 项目类别:
Standard Grant
EAGER: Tadoop: A Dual-Purpose Framework Taming the Bipolarity of Storage and Communication for High-Performance Computing and Data Analytics
EAGER:Tadoop:一个双用途框架,克服存储和通信的两极性,实现高性能计算和数据分析
- 批准号:
1432892 - 财政年份:2014
- 资助金额:
$ 22.5万 - 项目类别:
Standard Grant
CSR: Small: XooMR: Cross-Layer and Cross-Phase Cooperation for Fair and Efficient MapReduce
CSR:小:XooMR:跨层跨阶段合作实现公平高效的 MapReduce
- 批准号:
1320016 - 财政年份:2013
- 资助金额:
$ 22.5万 - 项目类别:
Standard Grant
II-New: A Compute and Storage Cluster for Multidisciplinary Research on Computer Systems and Scientific Simulations
II-New:用于计算机系统和科学模拟多学科研究的计算和存储集群
- 批准号:
1059376 - 财政年份:2011
- 资助金额:
$ 22.5万 - 项目类别:
Standard Grant
相似国自然基金
水凝胶改性陶瓷人工关节牢固结合界面的构筑与减磨润滑机理研究
- 批准号:
- 批准年份:2025
- 资助金额:0.0 万元
- 项目类别:省市级项目
锆酸铅基反铁电体畴动力学及其调控机理研究
- 批准号:
- 批准年份:2025
- 资助金额:0.0 万元
- 项目类别:省市级项目
载铁生物炭对土壤镉污染的吸附固定及微生物协同作用机制研究
- 批准号:
- 批准年份:2025
- 资助金额:0.0 万元
- 项目类别:省市级项目
SREBP转录因子BbSre1负调控球孢白僵菌抗真菌物质产生的机制研究
- 批准号:
- 批准年份:2025
- 资助金额:0.0 万元
- 项目类别:省市级项目
面向截肢患者运动感知重建的肌电假肢手关节运动反馈时变编码研究
- 批准号:
- 批准年份:2025
- 资助金额:0.0 万元
- 项目类别:省市级项目
面向水质应急快检的碳点/微流控限域增强发光传感研究
- 批准号:
- 批准年份:2025
- 资助金额:0.0 万元
- 项目类别:省市级项目
面向挠性压电太阳翼的物理信息混合建模与非同位控制方法研究
- 批准号:
- 批准年份:2025
- 资助金额:0.0 万元
- 项目类别:省市级项目
随机3维 Burgers 方程正则性研究
- 批准号:
- 批准年份:2025
- 资助金额:0.0 万元
- 项目类别:省市级项目
犬尿氨酸通过AhR/STAT3轴活化粒细胞样MDSCs促进慢性肾脏病心脏纤维化的机制研究
- 批准号:
- 批准年份:2025
- 资助金额:0.0 万元
- 项目类别:省市级项目
磁性的机器学习研究: 以图神经网络为中心
- 批准号:
- 批准年份:2025
- 资助金额:0.0 万元
- 项目类别:省市级项目
相似海外基金
Collaborative Research: OAC Core: Distributed Graph Learning Cyberinfrastructure for Large-scale Spatiotemporal Prediction
合作研究:OAC Core:用于大规模时空预测的分布式图学习网络基础设施
- 批准号:
2403312 - 财政年份:2024
- 资助金额:
$ 22.5万 - 项目类别:
Standard Grant
Collaborative Research: OAC CORE: Federated-Learning-Driven Traffic Event Management for Intelligent Transportation Systems
合作研究:OAC CORE:智能交通系统的联邦学习驱动的交通事件管理
- 批准号:
2414474 - 财政年份:2024
- 资助金额:
$ 22.5万 - 项目类别:
Standard Grant
Collaborative Research: OAC Core: Large-Scale Spatial Machine Learning for 3D Surface Topology in Hydrological Applications
合作研究:OAC 核心:水文应用中 3D 表面拓扑的大规模空间机器学习
- 批准号:
2414185 - 财政年份:2024
- 资助金额:
$ 22.5万 - 项目类别:
Standard Grant
Collaborative Research: OAC Core: Learning AI Surrogate of Large-Scale Spatiotemporal Simulations for Coastal Circulation
合作研究:OAC Core:学习沿海环流大规模时空模拟的人工智能替代品
- 批准号:
2402947 - 财政年份:2024
- 资助金额:
$ 22.5万 - 项目类别:
Standard Grant
Collaborative Research: OAC Core: Distributed Graph Learning Cyberinfrastructure for Large-scale Spatiotemporal Prediction
合作研究:OAC Core:用于大规模时空预测的分布式图学习网络基础设施
- 批准号:
2403313 - 财政年份:2024
- 资助金额:
$ 22.5万 - 项目类别:
Standard Grant
Collaborative Research: OAC Core: Learning AI Surrogate of Large-Scale Spatiotemporal Simulations for Coastal Circulation
合作研究:OAC Core:学习沿海环流大规模时空模拟的人工智能替代品
- 批准号:
2402946 - 财政年份:2024
- 资助金额:
$ 22.5万 - 项目类别:
Standard Grant
Collaborative Research: OAC Core: CropDL - Scheduling and Checkpoint/Restart Support for Deep Learning Applications on HPC Clusters
合作研究:OAC 核心:CropDL - HPC 集群上深度学习应用的调度和检查点/重启支持
- 批准号:
2403088 - 财政年份:2024
- 资助金额:
$ 22.5万 - 项目类别:
Standard Grant
Collaborative Research: OAC Core: CropDL - Scheduling and Checkpoint/Restart Support for Deep Learning Applications on HPC Clusters
合作研究:OAC 核心:CropDL - HPC 集群上深度学习应用的调度和检查点/重启支持
- 批准号:
2403090 - 财政年份:2024
- 资助金额:
$ 22.5万 - 项目类别:
Standard Grant
Collaborative Research: OAC: Core: Harvesting Idle Resources Safely and Timely for Large-scale AI Applications in High-Performance Computing Systems
合作研究:OAC:核心:安全及时地收集闲置资源,用于高性能计算系统中的大规模人工智能应用
- 批准号:
2403399 - 财政年份:2024
- 资助金额:
$ 22.5万 - 项目类别:
Standard Grant
Collaborative Research: OAC: Core: Harvesting Idle Resources Safely and Timely for Large-scale AI Applications in High-Performance Computing Systems
合作研究:OAC:核心:安全及时地收集闲置资源,用于高性能计算系统中的大规模人工智能应用
- 批准号:
2403398 - 财政年份:2024
- 资助金额:
$ 22.5万 - 项目类别:
Standard Grant