Design and implementation of task scheduling strategies for massive remote sensing data processing across multiple data centers
Design and implementation of task scheduling strategies for massive remote sensing data processing across multiple data centers
复制标题
DOI：
10.1002/spe.2229
复制
发表时间：
2014-07
期刊：
影响因子：
--
通讯作者：
Wanfeng Zhang;Lizhe Wang;Yan Ma;Dingsheng Liu
中科院分区：
文献类型：
--
作者：
Wanfeng Zhang;Lizhe Wang;Yan Ma;Dingsheng Liu
随着计算机技术和网络技术的发展，遥感数据处理的数据密集型应用越来越广泛。特别是在广泛分布的计算环境中，具有大量共享输入文件的任务包（BoTs）应用程序和具有数据依赖性的有向无环图（DAG）应用程序带来了新的挑战。本文提出了一种基于超图（hypergraph， PGH）的组分区策略，以建立文件共享模型。在PGH算法中，BoTs应用程序将被划分为几个组，以尽量减少数据传输的时间。我们还采用了另一种调度策略，即优化任务树（OTT）策略来处理具有数据依赖性的海量遥感数据处理DAG工作流。DAG任务的调度队列将根据优先级的变化进行更新。在GridSim仿真环境下，我们设计了调度程序中的Gridlets来测试PGH和OTT的性能。版权所有©2013 John Wiley & Sons， Ltd
Data intensive applications of remote sensing data processing are more and more widespread resulting from the evolutions in computer and network technologies. Especially, bags‐of‐tasks (BoTs) applications with a mass of sharing input files and directed acyclic graph (DAG) applications with data dependencies in a widely distributed computing environment bring new challenges. In this article, a strategy of partitioning group based on hypergraph (PGH) is introduced to formulate the model of sharing files. Within the PGH algorithm, BoTs applications would be partitioned into several groups to minimize the time of data transferring. We also adopted another scheduling policy, which is called optimized task tree (OTT) strategy to handle the DAG workflow of massive remote sensing data processing with data dependencies. A scheduling queue of DAG tasks would be updated according to the priorities changing. With the help of GridSim simulation environment, we designed the Gridlets within scheduler to test the performance of PGH and OTT. Copyright © 2013 John Wiley & Sons, Ltd.