AccPar: Tensor Partitioning for Heterogeneous Deep Learning Accelerators
AccPar: Tensor Partitioning for Heterogeneous Deep Learning Accelerators
复制标题
DOI：
10.1109/hpca47549.2020.00036
复制
发表时间：
2020-02
期刊：
影响因子：
--
通讯作者：
Linghao Song;Fan Chen;Youwei Zhuo;Xuehai Qian;H. Li;Yiran Chen
中科院分区：
文献类型：
--
作者：
Linghao Song;Fan Chen;Youwei Zhuo;Xuehai Qian;H. Li;Yiran Chen
深度神经网络（DNN）加速器作为特定领域架构的一个例子，在DNN推理方面取得了巨大成功。然而，同样重要的DNN训练的架构加速尚未得到充分研究。DNN的训练过程包括数据前向、误差后向和梯度计算，是一个更复杂的过程，具有更高的计算和通信强度。由于最近的研究表明，专业化回报递减，即“加速器墙”，我们认为，一个有前途的方法是探索多个性能限制的加速器之间的粗粒度并行，以支持DNN训练。然而，在多个异构加速器上分布计算以实现高吞吐量和平衡执行仍然具有挑战性。我们提出了AccPar，一个原则性和系统性的方法，确定张量之间的分配异构加速器阵列。与以前的经验或非系统的方法相比，AccPar考虑了完整的张量分区空间，可以揭示以前未知的新的并行配置。AccPar基于成本模型优化性能，该模型考虑了异构执行环境的计算和通信成本。因此，我们的方法可以避免现有的方法，使用通信作为代理的性能的缺点。AccPar中张量分区的增强灵活性允许在具有不同性能的加速器之间分配灵活的计算比率。所提出的搜索算法也适用于现代DNN（如ResNet）中新兴的多路径模式。我们在由TPU-v2和TPU-v3加速器组成的异构加速器阵列上模拟AccPar，用于训练大规模DNN模型，如Alexnet，Vgg系列和Resnet系列。最先进的“一个奇怪的技巧”（OWT）和HYPAR以及AccPar的平均性能改进，归一化到基线数据并行方案，其中每个加速器复制模型并并行处理不同的输入数据，分别为2.98倍，3.78倍和6.30倍。
Deep neural network (DNN) accelerators as an example of domain-specific architecture have demonstrated great success in DNN inference. However, the architecture acceleration for equally important DNN training has not yet been fully studied. With data forward, error backward and gradient calculation, DNN training is a more complicated process with higher computation and communication intensity. Because the recent research demonstrates a diminishing specialization return, namely, "accelerator wall", we believe that a promising approach is to explore coarse-grained parallelism among multiple performance-bounded accelerators to support DNN training. Distributing computations on multiple heterogeneous accelerators to achieve high throughput and balanced execution, however, remaining challenging. We present AccPar, a principled and systematic method of determining the tensor partition among heterogeneous accelerator arrays. Compared to prior empirical or unsystematic methods, AccPar considers the complete tensor partition space and can reveal previously unknown new parallelism configurations. AccPar optimizes the performance based on a cost model that takes into account both computation and communication costs of a heterogeneous execution environment. Hence, our method can avoid the drawbacks of existing approaches that use communication as a proxy of the performance. The enhanced flexibility of tensor partitioning in AccPar allows the flexible ratio of computations to be distributed among accelerators with different performances. The proposed search algorithm is also applicable to the emerging multi-path patterns in modern DNNs such as ResNet. We simulate AccPar on a heterogeneous accelerator array composed of both TPU-v2 and TPU-v3 accelerators for the training of large-scale DNN models such as Alexnet, Vgg series, and Resnet series. The average performance improvements of the state-of-the-art "one weird trick" (OWT) and HYPAR, and AccPar, normalized to the baseline data parallelism scheme where each accelerator replicates the model and processes different input data in parallel, are 2.98×, 3.78×, and 6.30×, respectively.