Heterogeneous MacroTasking (HeMT) for Parallel Processing in the Cloud
Heterogeneous MacroTasking (HeMT) for Parallel Processing in the Cloud
复制标题
DOI：
10.1145/3429885.3429962
复制
发表时间：
2020-12
期刊：
影响因子：
--
通讯作者：
Y. Shan;G. Kesidis;Aman Jain;B. Urgaonkar;J. Khamse-Ashari;I. Lambadaris
中科院分区：
文献类型：
--
作者：
Y. Shan;G. Kesidis;Aman Jain;B. Urgaonkar;J. Khamse-Ashari;I. Lambadaris
在并行计算系统中，使用微任务(微任务)一直被认为是一种有效的负载平衡方法。当与容器执行节点在空闲时拉入工作相结合时，微任务处理具有理想的特性，即自动调整其负载分配以适应参与节点的处理能力--更强大的节点更快地完成其工作，因此，更快地拉入额外的工作。因此，微任务处理被认为特别适用于处理能力不同且工作负载特性不佳的环境。然而，微任务处理确实有额外的调度和I/O开销，这在某些情况下可能会使其成本高昂。此外，通常需要学习最佳任务大小。本文研究了一种可供选择的负载均衡方案--异构宏任务(HEMT)，其中根据节点的处理能力有意地对工作负载进行倾斜。我们在ApacheSpark应用程序框架中实现并开源了一个HEMT原型，并使用ApacheMesos集群管理器进行了实验。实验表明，当学习到节点处理能力的特定工作负载估计时，Spark with HEMT与基线同构微任务(HomT)系统相比，为实际的多阶段数据处理工作负载提供了高达10%的平均完成时间。
Using tiny tasks (microtasks) has long been regarded an effective way of load balancing in parallel computing systems. When combined with containerized execution nodes pulling in work upon becoming idle, microtasking has the desirable property of automatically adapting its load distribution to the processing capacities of participating nodes - more powerful nodes finish their work sooner and, therefore, pull in additional work faster. As a result, microtasking is deemed especially desirable in settings with heterogeneous processing capacities and poorly characterized workloads. However, microtasking does have additional scheduling and I/O overheads that may make it costly in some scenarios. Moreover, the optimal task size generally needs to be learned. We herein study an alternative load balancing scheme - Heterogeneous MacroTasking (HeMT) - wherein workload is intentionally skewed according to the nodes' processing capacity. We implemented and open-sourced a prototype of HeMT within the Apache Spark application framework and conducted experiments using the Apache Mesos cluster manager. It's shown experimentally that when workload-specific estimates of nodes' processing capacities are learned, Spark with HeMT offers up to 10% shorter average completion times for realistic, multistage data-processing workloads over the baseline Homogeneous microTasking (HomT) system.