Using High-Bandwidth Networks Efficiently for Fast Graph Computation
Using High-Bandwidth Networks Efficiently for Fast Graph Computation
复制标题
有效使用高带宽网络进行快速图计算
DOI：
10.1109/tpds.2018.2875084
复制
发表时间：
2019-05-01
影响因子：
5.3
通讯作者：
Wu, Yunxiang
中科院分区：
文献类型：
--
作者：
Cheng, Yongli;Jiang, Hong;Wu, Yunxiang
如今，高带宽网络比以往任何时候都更容易接入。然而，现有的分布式图处理框架，如GPS，由于其低效的计算和通信模型，无法有效地利用这些网络中的额外带宽容量来获得更高的性能，导致用户对图计算结果的等待时间很长。根本原因在于这些框架的计算和通信模型生成、发送和接收消息的速度非常慢，以至于只利用了可用网络带宽的一小部分。在本文中，我们提出了一个高性能的分布式图形处理框架，称为BlitzG，来解决这个问题。该框架充分利用了可用的网络带宽容量进行快速图形处理。我们的方法旨在通过使用一个新的精简的以顶点为中心的计算模型来显著减少(i)每个顶点的计算工作量，以实现快速消息生成；（ii）通过设计一个轻量级的以消息为中心的通信模型来实现快速消息传递的平均消息开销。在40Gbps以太网上进行的评估，由真实世界的图形数据集驱动，表明BlitzG比GPS性能高出27倍，平均为20.7倍。
Nowadays, high-bandwidth networks are more easily accessible than ever before. However, existing distributed graph-processing frameworks, such as GPS, fail to efficiently utilize the additional bandwidth capacity in these networks for higher performance, due to their inefficient computation and communication models, leading to very long waiting times experienced by users for the graph-computing results. The root cause lies in the fact that the computation and communication models of these frameworks generate, send and receive messages so slowly that only a small fraction of the available network bandwidth is utilized. In this paper, we propose a high-performance distributed graph-processing framework, called BlitzG, to address this problem. This framework fully exploits the available network bandwidth capacity for fast graph processing. Our approach aims at significant reduction in (i) the computation workload of each vertex for fast message generation by using a new slimmed-down vertex-centric computation model and (ii) the average message overhead for fast message delivery by designing a light-weight message-centric communication model. Evaluation on a 40Gbps Ethernet, driven by real-world graph datasets, shows that BlitzG outperforms GPS by up to 27x with an average of 20.7x.