Libra and the Art of Task Sizing in Big-Data Analytic Systems
Libra and the Art of Task Sizing in Big-Data Analytic Systems
复制标题
DOI：
10.1145/3357223.3362720
复制
发表时间：
2019-11
期刊：
影响因子：
--
通讯作者：
Ruikang Li;Peizhen Guo;Bo Hu;Wenjun Hu
中科院分区：
文献类型：
--
作者：
Ruikang Li;Peizhen Guo;Bo Hu;Wenjun Hu
尽管在数据密集型计算框架中对作业调度进行了广泛的研究，但很少考虑优化作业划分以提高资源利用率和处理效率。相反，分区和作业规模是一种黑暗艺术，通常留给开发人员的直觉和试错风格的实验。在这项工作中，我们建议，正如作业调度和资源分配外包给一个可信的机制外部的工作量，所以也应该是分区数据的责任作为一个决定因素的任务大小。作业分区本质上涉及确定分区大小，以最细的粒度匹配资源分配。这是一个复杂的，多维的问题，是高度特定于应用程序：资源分配，计算运行时间，洗牌和减少通信要求，任务启动开销都有很大的影响，最有效的任务规模，以有效的处理。根据分区大小的不同，作业完成时间可能会相差10倍之多！幸运的是，我们观察到一个总的趋势，在不同的设置之间的充分资源利用率和系统开销之间的权衡。最佳作业分区大小平衡了这两种相互冲突的力量。鉴于这种趋势，我们设计Libra来自动化作业划分作为框架扩展。我们将Libra与Spark集成，并评估其在EC2上的性能。与最先进的技术相比，Libra可以将单个作业的执行时间减少25%到70%。
Despite extensive investigation of job scheduling in data-intensive computation frameworks, less consideration has been given to optimizing job partitioning for resource utilization and efficient processing. Instead, partitioning and job sizing are a form of dark art, typically left to developer intuition and trial-and-error style experimentation. In this work, we propose that just as job scheduling and resource allocation are out-sourced to a trusted mechanism external to the workload, so too should be the responsibility for partitioning data as a determinant for task size. Job partitioning essentially involves determining the partition sizes to match the resource allocation at the finest granularity. This is a complex, multi-dimensional problem that is highly application specific: resource allocation, computational runtime, shuffle and reduce communication requirements, and task startup overheads all have strong influence on the most effective task size for efficient processing. Depending on the partition size, the job completion time can differ by as much as 10 times! Fortunately, we observe a general trend underlying the tradeoff between full resource utilization and system overhead across different settings. The optimal job partition size balances these two conflicting forces. Given this trend, we design Libra to automate job partitioning as a framework extension. We integrate Libra with Spark and evaluate its performance on EC2. Compared to state-of-the-art techniques, Libra can reduce the individual job execution time by 25% to 70%.