Queue Waiting Time Aware Dynamic Workflow Scheduling in Multicluster Environments
Queue Waiting Time Aware Dynamic Workflow Scheduling in Multicluster Environments
复制标题
DOI：
10.1007/s11390-010-9371-8
复制
发表时间：
2010-07
影响因子：
0.7
通讯作者：
Zhifeng Yu;Weisong Shi
中科院分区：
文献类型：
--
作者：
Zhifeng Yu;Weisong Shi
工作流在科学计算中盛行。多集群环境的出现并提供了更多的资源，这有利于工作流程，但也对传统的工作流程调度启发式提出了挑战。在多集群环境中，每个集群都有自己独立的工作负载管理系统。作业在执行之前会排队，如果分派到不同的集群，它们会经历不同的资源可用性和等待时间。然而，现有的调度启发式既不考虑队列等待时间，也不平衡性能增益与数据移动成本。所提出的算法利用了队列等待时间预测技术的进步，并根据经验研究资源需求的可调性是否有助于调度。对真实工作负载跟踪和测试平台进行的广泛实验表明，队列等待时间感知算法将平均完成时间方面的工作流程性能提高了 3 到 10 倍，并且数据移动成本相对较低。
Workflows are prevailing in scientific computation. Multicluster environments emerge and provide more resources, benefiting workflows but also challenging the traditional workflow scheduling heuristics. In a multicluster environment, each cluster has its own independent workload management system. Jobs are queued up before getting executed, they experience different resource availability and wait time if dispatched to different clusters. However, existing scheduling heuristics neither consider the queue wait time nor balance the performance gain with data movement cost. The proposed algorithm leverages the advancement of queue wait time prediction techniques and empirically studies if the tunability of resource requirements helps scheduling. The extensive experiment with both real workload traces and test bench shows that the queue wait time aware algorithm improves workflow performance by 3 to 10 times in terms of average makespan with relatively very low cost of data movement.