Traffic-aware task placement with guaranteed job completion time for geo-distributed big data
Traffic-aware task placement with guaranteed job completion time for geo-distributed big data
复制标题
DOI：
10.1109/icc.2017.7996541
复制
发表时间：
2017-05
期刊：
影响因子：
--
通讯作者：
Peng Li;T. Miyazaki;Song Guo
中科院分区：
文献类型：
--
作者：
Peng Li;T. Miyazaki;Song Guo
大数据分析通常被转化为在地理分布式数据中心上运行的并行作业。与单一数据中心不同，地理分布式环境由于位于不同区域的数据中心之间的网络带宽有限，给大数据分析带来了巨大的挑战。尽管研究工作一直致力于地理分布式大数据，但由于其性能欠佳或复杂性较高，其结果仍远未达到高效。在本文中，我们提出了一种流量感知任务放置，以最大限度地减少大数据作业的作业完成时间。我们将该问题表述为非凸优化问题，并设计一种算法来解决该问题，并证明其性能差距。最后，进行广泛的模拟来评估我们建议的性能。模拟结果表明，与聚合所有数据进行集中处理的传统方法相比，我们的算法可以将作业完成时间缩短 40%。同时，它与最优解的性能差距只有10%，但解决问题的时间极短。
Big data analysis is usually casted into parallel jobs running on geo-distributed data centers. Different from a single data center, geo-distributed environment imposes big challenges for big data analytics due to the limited network bandwidth between data centers located in different regions. Although research efforts have been devoted to geo-distributed big data, the results are still far from being efficient because of their suboptimal performance or high complexity. In this paper, we propose a traffic-aware task placement to minimize job completion time of big data jobs. We formulate the problem as a non-convex optimization problem and design an algorithm to solve it with proved performance gap. Finally, extensive simulations are conducted to evaluate the performance of our proposal. The simulation results show that our algorithm can reduce job completion time by 40%, compared to a conventional approach that aggregates all data for centralized processing. Meanwhile, it has only 10% performance gap with the optimal solution, but its problem-solving time is extremely small.