Swallow: Joint Online Scheduling and Coflow Compression in Datacenter Networks
Swallow: Joint Online Scheduling and Coflow Compression in Datacenter Networks
复制标题
DOI：
10.1109/ipdps.2018.00060
复制
发表时间：
2018-05
期刊：
影响因子：
--
通讯作者：
Qihua Zhou;Peng Li;Kun Wang;Deze Zeng;Song Guo;M. Guo
中科院分区：
文献类型：
--
作者：
Qihua Zhou;Peng Li;Kun Wang;Deze Zeng;Song Guo;M. Guo
数据中心的大数据分析往往涉及到数据并行作业的调度，而数据中心网络带宽有限，这是数据中心分析的瓶颈。为了缓解带宽的短缺，一些现有的工作提出了流量压缩来减少网络上传输的数据量。然而，他们提出的流量压缩在作业级别以粗粒度的方式工作，为进一步的性能改进留下了很大的优化空间。在本文中，我们提出了一个流级流量压缩和调度系统，称为Swallow，以加速数据密集型应用程序。具体来说，我们的目标是coflows，这是对大数据作业生成的并行流的优雅抽象。以最小化共流完成时间（CCT）为目标，提出了一种启发式算法，称为最快容量-处置-优先（FVDV），并基于Spark实现了Swallow。跟踪驱动的仿真和实际实验结果表明，我们的系统优于现有的算法。与目前最有效的协同流调度算法之一Varys的SEBF相比，Swallow可以将CCT和作业完成时间（JCT）平均分别减少1.47倍和1.66倍。此外，通过coflow压缩，Swallow平均减少了48.41%的数据流量。
Big data analytics in datacenters often involves scheduling of data-parallel job, which are bottlenecked by limited bandwidth of datacenter networks. To alleviate the shortage of bandwidth, some existing work has proposed traffic compression to reduce the amount of data transmitted over the network. However, their proposed traffic compression works in a coarse-grained manner at job level, leaving a large optimization space unexplored for further performance improvement. In this paper, we propose a flow-level traffic compression and scheduling system, called Swallow, to accelerate data-intensive applications. Specifically, we target on coflows, which is an elegant abstraction of parallel flows generated by big data jobs. With the objective of minimizing coflow completion time (CCT), we propose a heuristic algorithm called Fastest-Volume-Disposal-First (FVDV) and implement Swallow based on Spark. The results of both trace-driven simulations and real experiments show the superiority of our system, over existing algorithms. Swallow can reduce CCT and job completion time (JCT) by up to 1.47 × and 1.66 × on average, respectively, over the SEBF in Varys, one of the most efficient coflow scheduling algorithms so far. Moreover, with coflow compression, Swallow reduces data traffic by up to 48.41% on average.