Efficient Coflow Scheduling with Varys
Efficient Coflow Scheduling with Varys
复制标题
DOI：
10.1145/2740070.2626315
复制
发表时间：
2014-10-01
影响因子：
2.8
通讯作者：
Stoica, Ion
中科院分区：
文献类型：
--
作者：
Chowdhury, Mosharaf;Zhong, Yuan;Stoica, Ion
数据并行应用程序中的通信通常涉及并行流的集合。传统的技术，以优化流级指标不执行优化这样的集合，因为网络在很大程度上是不可知的应用程序级的要求。最近提出的coflow抽象弥合了这一差距，为网络调度创造了新的机会。在本文中，我们解决两个不同的目标：减少数据密集型作业的通信时间和保证可预测的通信时间的同流间调度。介绍了具有耦合资源的并行开放车间调度问题，分析了其复杂性，并提出了有效的算法来优化任一目标。我们提出了瓦里斯，一个系统，使数据密集型框架使用coflows和建议的算法，同时保持高网络利用率和保证饥饿自由。EC2部署和跟踪驱动的模拟表明，通信阶段完成速度平均快3.16倍，与每个流机制相比，使用Varys的coflow达到其截止日期的速度高达2倍。此外，Varys的性能比非抢占式coflow路由器高出5倍以上
Communication in data-parallel applications often involves a collection of parallel flows. Traditional techniques to optimize flowlevel metrics do not perform well in optimizing such collections, because the network is largely agnostic to application-level requirements. The recently proposed coflow abstraction bridges this gap and creates new opportunities for network scheduling. In this paper, we address inter-coflow scheduling for two different objectives: decreasing communication time of data-intensive jobs and guaranteeing predictable communication time. We introduce the concurrent open shop scheduling with coupled resources problem, analyze its complexity, and propose effective heuristics to optimize either objective. We present Varys, a system that enables data-intensive frameworks to use coflows and the proposed algorithms while maintaining high network utilization and guaranteeing starvation freedom. EC2 deployments and trace-driven simulations show that communication stages complete up to 3.16x faster on average and up to 2x more coflows meet their deadlines using Varys in comparison to per-flow mechanisms. Moreover, Varys outperforms non-preemptive coflow schedulers by more than 5x