权益分类	功能权益	普通用户	{{item.name}}会员
{{category.name}}	{{benefitItem.name}}

CRII: SHF: Optimizing Deep Learning Training through Modeling and Scheduling Support

CRII：SHF：通过建模和调度支持优化深度学习训练

基本信息

批准号：
1756013
负责人：
Feng Yan
金额：
$ 17.5万
依托单位：
Board of Regents, NSHE, obo University of Nevada, Reno
依托单位国家：
美国
项目类别：
Standard Grant
财政年份：
2018
资助国家：
美国
起止时间：
2018-06-01 至 2021-05-31
项目状态：
已结题

来源：
https://www.nsf.gov/awardsearch/showAward?AWD_ID=1756013&HistoricalAwards=false
关键词：
CRII SHF Optimizing Deep Learning

项目摘要

Deep learning models trained on large amounts of data using lots of computing resources have recently achieved state-of-the-art training performance on important yet challenging artificial intelligence tasks. The success of deep learning has attracted significant research interest from hardware and software communities to improve training speed and efficiency. Despite the great efforts and rapid progress made, one important bridge to connect software and hardware support with deep learning domain knowledge is still missing: efficient configuration exploration and runtime scheduling. Both the quality of deep learning models and the training time are very sensitive to many adjustable parameters that are set before and during the training process, including the hyperparameter configurations (such as learning rate, momentum, number and size of hidden layers) and system configurations (such as thread parallelism, model parallelism, and data parallelism). Efficient exploration of hyperparameter configurations and judicious selection of system configurations is of great importance to find high-quality models with affordable time and cost. This is however a challenging problem due to a huge search space, expensive training runtime, sparsity of good configurations, and scarcity of time and resources.The objective of this research work is to systematically study the unique properties of deep learning systems and workloads, and establish new modeling and scheduling methodologies for improving deep learning training. The PI aims to improve the efficiency of discovering high performing models through a dynamic scheduling methodology driven by a novel hyperparameter configuration classification approach. The PI aims at developing an accuracy- and efficiency-aware hybrid scheduling methodology that makes judicious scheduling decisions based on a global view of both the time dimension (accuracy potential) and spatial dimension (efficiency potential) information. This research work integrates techniques in workload characterization, performance modeling, resource management, and scheduling to dramatically speedup the training process while significantly reducing the cost in time and resources. More broadly, this project will gain foundational knowledge about the interaction between software-hardware support and deep learning domain knowledge. This knowledge can help design next generation deep learning systems and frameworks, making deep learning training handy for researchers and practitioners with limited system and machine learning domain expertise. This research will help enhance curriculum and provide research topics for both undergraduate and graduate students, especially students from underrepresented groups.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

使用大量计算资源对大量数据进行训练的深度学习模型最近在重要但具有挑战性的人工智能任务上取得了最先进的训练性能。深度学习的成功吸引了硬件和软件界对提高训练速度和效率的浓厚研究兴趣。尽管已经取得了巨大的努力和快速的进展，但连接软件和硬件支持与深度学习领域知识的一个重要桥梁仍然缺失：高效的配置探索和运行时调度。深度学习模型的质量和训练时间对训练前和训练过程中设置的许多可调参数都非常敏感，包括超参数配置(如学习速度、动量、隐含层的数量和大小)和系统配置(如线程并行、模型并行、数据并行)。对超参数配置的有效探索和对系统配置的明智选择对于以负担得起的时间和成本找到高质量的模型至关重要。然而，这是一个具有挑战性的问题，因为搜索空间巨大，训练时间昂贵，良好的配置稀疏，时间和资源稀缺。本研究工作的目的是系统地研究深度学习系统和工作量的独特属性，并建立新的建模和调度方法来改进深度学习训练。PI旨在通过一种新的超参数配置分类方法驱动的动态调度方法来提高发现高性能模型的效率。PI旨在开发一种精度和效率感知的混合调度方法，该方法基于时间维度(精度潜力)和空间维度(效率潜力)信息的全局视图做出明智的调度决策。这项研究工作集成了工作负载表征、性能建模、资源管理和调度方面的技术，以显著加快培训过程，同时显著降低时间和资源成本。更广泛地说，这个项目将获得关于软硬件支持和深度学习领域知识之间相互作用的基础知识。这些知识可以帮助设计下一代深度学习系统和框架，使深度学习培训对于系统和机器学习领域专业知识有限的研究人员和从业者来说非常方便。这项研究将有助于加强课程，并为本科生和研究生提供研究课题，特别是来自代表性不足群体的学生。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

期刊论文数量（37）

专著数量（0）

科研奖励数量（0）

会议论文数量（0）

专利数量（0）

CEDULE: A Scheduling Framework for Burstable Performance in Cloud Computing

CEDULE：云计算中突发性能的调度框架

DOI：
10.1109/icac.2018.00024
发表时间：
2018
期刊：
2018 IEEE International Conference on Autonomic Computing (ICAC
影响因子：
0
作者：
Ali, Ahsan;Pinciroli, Riccardo;Yan, Feng;Smirni, Evgenia
通讯作者：
Smirni, Evgenia

InfiniCache: Exploiting Ephemeral Serverless Functions to Build a Cost-Effective Memory Cache

DOI：
发表时间：
2020-01
期刊：
影响因子：
0
作者：
Ao Wang;Jingyuan Zhang;Xiaolong Ma;Ali Anwar;Lukas Rupprecht;Dimitrios Skourtis;Vasily Tarasov
通讯作者：
Ao Wang;Jingyuan Zhang;Xiaolong Ma;Ali Anwar;Lukas Rupprecht;Dimitrios Skourtis;Vasily Tarasov

BatchCrypt: Efficient Homomorphic Encryption for Cross-Silo Federated Learning

DOI：
发表时间：
2020
期刊：
影响因子：
0
作者：
Chengliang Zhang;Suyi Li;Junzhe Xia;Wei Wang;Feng Yan;Yang Liu
通讯作者：
Chengliang Zhang;Suyi Li;Junzhe Xia;Wei Wang;Feng Yan;Yang Liu

Gradient Compression Supercharged High-Performance Data Parallel DNN Training

DOI：
10.1145/3477132.3483553
发表时间：
2021-10
期刊：
Proceedings of the ACM SIGOPS 28th Symposium on Operating Systems Principles
影响因子：
0
作者：
Youhui Bai;Cheng Li;Quan Zhou;Jun Yi;Ping Gong;Feng Yan;Ruichuan Chen;Yinlong Xu
通讯作者：
Youhui Bai;Cheng Li;Quan Zhou;Jun Yi;Ping Gong;Feng Yan;Ruichuan Chen;Yinlong Xu

It's not a Sprint, it's a Marathon: Stretching Multi-resource Burstable Performance in Public Clouds

这不是冲刺，而是马拉松：在公共云中扩展多资源突发性能

DOI：
10.1145/3366626.3368130
发表时间：
2019
期刊：
Proceedings of the 20th International Middleware Conference (Middleware 2019
影响因子：
0
作者：
Ali, Ahsan;Pinciroli, Riccardo;Yan, Feng;Smirni, Evgenia
通讯作者：
Smirni, Evgenia