Performance and Consistency Analysis for Distributed Deep Learning Applications
Performance and Consistency Analysis for Distributed Deep Learning Applications
复制标题
DOI：
10.1109/ipccc50635.2020.9391566
复制
发表时间：
2020-11
期刊：
影响因子：
--
通讯作者：
Danlin Jia;M. Saha;J. Bhimani;N. Mi
中科院分区：
文献类型：
--
作者：
Danlin Jia;M. Saha;J. Bhimani;N. Mi
加速深度神经网络（DNN）模型的训练对于在计算机视觉和语音识别等领域成功使用深度学习技术非常重要。分布式框架有助于加快大型DNN模型和数据集的训练过程。在对卷积神经网络（CNN）的计算过程进行数学分析的基础上，为提高模型的精度和训练效率做了大量的工作。然而，要在真实的世界中运行分布式深度学习应用程序，用户和开发人员需要考虑系统资源分布的影响。在这项工作中，我们部署了一个具有多个虚拟机的真实的分布式深度学习集群。我们进行了深入的分析，以了解系统配置，分布类型和应用程序参数对分布式深度学习应用程序的延迟和正确性的影响。通过分析运行时系统利用率和跟踪应用程序活动，分析了不同模型一致性和数据并行性下的性能差异。根据我们的观察和分析，我们制定了设计指南，用于加速虚拟化环境中的分布式深度学习训练。
Accelerating the training of Deep Neural Network (DNN) models is very important for successfully using deep learning techniques in fields like computer vision and speech recognition. Distributed frameworks help to speed up the training process for large DNN models and datasets. Plenty of works have been done to improve model accuracy and training efficiency, based on mathematical analysis of computations in the Con-volutional Neural Networks (CNN). However, to run distributed deep learning applications in the real world, users and developers need to consider the impacts of system resource distribution. In this work, we deploy a real distributed deep learning cluster with multiple virtual machines. We conduct an in-depth analysis to understand the impacts of system configurations, distribution typologies, and application parameters, on the latency and correctness of the distributed deep learning applications. We analyze the performance diversity under different model consistency and data parallelism by profiling run-time system utilization and tracking application activities. Based on our observations and analysis, we develop design guidelines for accelerating distributed deep-learning training on virtualized environments.