Beyond Data and Model Parallelism for Deep Neural Networks
Beyond Data and Model Parallelism for Deep Neural Networks
复制标题
DOI：
--
复制
发表时间：
2018-07
期刊：
影响因子：
--
通讯作者：
Zhihao Jia;M. Zaharia;A. Aiken
中科院分区：
文献类型：
--
作者：
Zhihao Jia;M. Zaharia;A. Aiken
训练深度神经网络（dnn）的计算需求已经增长到现在并行训练是标准实践的程度。现有的深度学习系统通常使用数据或模型并行，但不幸的是，这些策略经常导致次优的并行化性能。在本文中，我们为深度神经网络定义了一个更全面的并行化策略搜索空间，称为SOAP，其中包括在样本、操作、属性和参数维度上并行化深度神经网络的策略。我们还提出了FlexFlow，这是一个深度学习框架，它使用SOAP空间的引导随机搜索来为特定的并行机器找到快速并行化策略。为了加速这种搜索，FlexFlow引入了一种新的执行模拟器，它可以准确地预测并行化策略的性能，并且比之前必须执行每种策略的方法快三个数量级。我们在两个GPU集群上用六个真实世界的DNN基准测试来评估FlexFlow，结果表明，FlexFlow可以比最先进的方法提高3.8倍的训练吞吐量，即使包括其搜索时间，也可以提高可扩展性。
The computational requirements for training deep neural networks (DNNs) have grown to the point that it is now standard practice to parallelize training. Existing deep learning systems commonly use data or model parallelism, but unfortunately, these strategies often result in suboptimal parallelization performance. In this paper, we define a more comprehensive search space of parallelization strategies for DNNs called SOAP, which includes strategies to parallelize a DNN in the Sample, Operation, Attribute, and Parameter dimensions. We also propose FlexFlow, a deep learning framework that uses guided randomized search of the SOAP space to find a fast parallelization strategy for a specific parallel machine. To accelerate this search, FlexFlow introduces a novel execution simulator that can accurately predict a parallelization strategy's performance and is three orders of magnitude faster than prior approaches that have to execute each strategy. We evaluate FlexFlow with six real-world DNN benchmarks on two GPU clusters and show that FlexFlow can increase training throughput by up to 3.8x over state-of-the-art approaches, even when including its search time, and also improves scalability.