Characterizing Deep Learning over Big Data (DLoBD) Stacks on RDMA-Capable Networks
Characterizing Deep Learning over Big Data (DLoBD) Stacks on RDMA-Capable Networks
复制标题
在支持 RDMA 的网络上表征大数据深度学习 (DLoBD) 堆栈
DOI：
--
复制
发表时间：
2017
期刊：
影响因子：
--
通讯作者：
D. Panda
中科院分区：
文献类型：
--
作者：
Xiaoyi Lu;Haiyang Shi;M. Javed;Rajarshi Biswas;D. Panda
大数据深度学习（DLoBD）正在成为从大量收集的数据中挖掘价值的最重要的研究范式之一。许多新兴的深度学习框架开始在大数据堆栈上运行，例如 Hadoop 和 Spark。随着 HPC、大数据和深度学习的融合，这些 DLoBD 堆栈正在利用 RDMA 和基于多核/众核的 CPU/GPU。尽管该领域正在进行很多活动，但缺乏系统研究来分析支持 RDMA 的网络和 CPU/GPU 对 DLoBD 堆栈的影响。为了填补这一空白，我们提出了一种系统的表征方法，并对三个代表性的 DLoBD 堆栈（即 CaffeOnSpark、TensorFlowOnSpark 和 BigDL）进行了广泛的性能评估，以揭示有关性能、可扩展性、准确性和资源利用率的有趣趋势。我们的观察表明，与基于 IPoIB 的方案相比，基于 RDMA 的 DLoBD 堆栈设计可实现高达 2.7 倍的加速。与 InfiniBand 集群上的 IPoIB 方案相比，RDMA 方案还可以更好地扩展并更有效地利用资源。在大多数情况下，基于 GPU 的深度学习可以优于基于 CPU 的设计，但并非总是如此。我们看到，对于 MNIST 上的 LeNet，CPU + MKL 在 16 个节点上可以取得比 GPU 和 GPU + cuDNN 更好的性能。通过我们的评估，我们发现当前一代 DLoBD 堆栈的设计还有很大的改进空间。
Deep Learning over Big Data (DLoBD) is becoming one of the most important research paradigms to mine value from the massive amount of gathered data. Many emerging deep learning frameworks start running over Big Data stacks, such as Hadoop and Spark. With the convergence of HPC, Big Data, and Deep Learning, these DLoBD stacks are taking advantage of RDMA and multi-/many-core based CPUs/GPUs. Even though a lot of activities are happening in the field, there is a lack of systematic studies on analyzing the impact of RDMA-capable networks and CPU/GPU on DLoBD stacks. To fill this gap, we propose a systematical characterization methodology and conduct extensive performance evaluations on three representative DLoBD stacks (i.e., CaffeOnSpark, TensorFlowOnSpark, and BigDL) to expose the interesting trends regarding performance, scalability, accuracy, and resource utilization. Our observations show that RDMA-based design for DLoBD stacks can achieve up to 2.7x speedup compared to the IPoIB based scheme. The RDMA scheme can also scale better and utilize resources more efficiently than the IPoIB scheme over InfiniBand clusters. For most cases, GPU-based deep learning can outperform CPU-based designs, but not always. We see that for LeNet on MNIST, CPU + MKL can achieve better performance than GPU and GPU + cuDNN on 16 nodes. Through our evaluation, we see that there are large rooms to improve the designs of current generation DLoBD stacks further.