CRII: SHF: Optimizing Deep Learning Training through Modeling and Scheduling Support

使用大量计算资源在大量数据上训练的深度学习模型最近在重要但具有挑战性的人工智能任务上取得了最先进的训练性能。深度学习的成功吸引了硬件和软件社区的极大研究兴趣，以提高训练速度和效率。尽管付出了巨大的努力并取得了快速的进展，但仍然缺少一个将软件和硬件支持与深度学习领域知识联系起来的重要桥梁：高效的配置探索和运行时调度。深度学习模型的质量和训练时间对训练过程之前和期间设置的许多可调参数非常敏感，包括超参数配置（如学习率，动量，隐藏层的数量和大小）和系统配置（如线程并行性，模型并行性和数据并行性）。有效地探索超参数配置和明智地选择系统配置对于以可承受的时间和成本找到高质量的模型非常重要。然而，由于搜索空间巨大，训练时间昂贵，良好配置稀疏，时间和资源稀缺，这是一个具有挑战性的问题。本研究工作的目标是系统地研究深度学习系统和工作负载的独特属性，并建立新的建模和调度方法来改进深度学习训练。PI旨在通过一种新的超参数配置分类方法驱动的动态调度方法来提高发现高性能模型的效率。PI的目的是开发一个准确性和效率意识的混合调度方法，使明智的调度决策的基础上的时间维度（准确性潜力）和空间维度（效率潜力）的信息的全局视图。这项研究工作集成了工作负载表征，性能建模，资源管理和调度技术，大大加快了培训过程，同时显着降低了时间和资源的成本。更广泛地说，该项目将获得有关软硬件支持和深度学习领域知识之间相互作用的基础知识。这些知识可以帮助设计下一代深度学习系统和框架，使深度学习培训对于系统和机器学习领域专业知识有限的研究人员和从业者来说非常方便。这项研究将有助于提高课程，并为本科生和研究生提供研究课题，特别是来自代表性不足的群体的学生。该奖项反映了NSF的法定使命，并已被认为是值得通过使用基金会的智力价值和更广泛的影响审查标准进行评估的支持。

Deep learning models trained on large amounts of data using lots of computing resources have recently achieved state-of-the-art training performance on important yet challenging artificial intelligence tasks. The success of deep learning has attracted significant research interest from hardware and software communities to improve training speed and efficiency. Despite the great efforts and rapid progress made, one important bridge to connect software and hardware support with deep learning domain knowledge is still missing: efficient configuration exploration and runtime scheduling. Both the quality of deep learning models and the training time are very sensitive to many adjustable parameters that are set before and during the training process, including the hyperparameter configurations (such as learning rate, momentum, number and size of hidden layers) and system configurations (such as thread parallelism, model parallelism, and data parallelism). Efficient exploration of hyperparameter configurations and judicious selection of system configurations is of great importance to find high-quality models with affordable time and cost. This is however a challenging problem due to a huge search space, expensive training runtime, sparsity of good configurations, and scarcity of time and resources.The objective of this research work is to systematically study the unique properties of deep learning systems and workloads, and establish new modeling and scheduling methodologies for improving deep learning training. The PI aims to improve the efficiency of discovering high performing models through a dynamic scheduling methodology driven by a novel hyperparameter configuration classification approach. The PI aims at developing an accuracy- and efficiency-aware hybrid scheduling methodology that makes judicious scheduling decisions based on a global view of both the time dimension (accuracy potential) and spatial dimension (efficiency potential) information. This research work integrates techniques in workload characterization, performance modeling, resource management, and scheduling to dramatically speedup the training process while significantly reducing the cost in time and resources. More broadly, this project will gain foundational knowledge about the interaction between software-hardware support and deep learning domain knowledge. This knowledge can help design next generation deep learning systems and frameworks, making deep learning training handy for researchers and practitioners with limited system and machine learning domain expertise. This research will help enhance curriculum and provide research topics for both undergraduate and graduate students, especially students from underrepresented groups.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.