A speculative approach to spatial-temporal efficiency with multi-objective optimization in a heterogeneous cloud environment
A speculative approach to spatial-temporal efficiency with multi-objective optimization in a heterogeneous cloud environment
复制标题
异构云环境中多目标优化的时空效率推测方法
DOI：
10.1002/sec.1582
复制
发表时间：
2016-11-25
影响因子：
--
通讯作者：
Linge, Nigel
中科院分区：
文献类型：
--
作者：
Liu, Qi;Cai, Weidong;Linge, Nigel
一个异构的云系统，例如Hadoop 2.6.0平台，提供分布式但有凝聚力的服务，具有丰富的大规模管理，可靠性和容错功能。在大数据处理方面，新建的云集群面临着性能优化的挑战，重点是更快的任务执行和更有效地使用计算资源。目前提出的方法集中在时间上的改进，即缩短MapReduce时间，但很少关注存储占用;然而，不平衡的云存储策略可能会耗尽那些MapReduce周期较长的节点，并进一步挑战整个集群的安全性和稳定性。本文针对异构云环境下的时空效率问题，提出了一种自适应方法。提出了一种基于优化核函数的极限学习机算法的预测模型，用于快速预测作业执行时间和空间占用情况，从而通过一种多目标的时空优化NSGA-Ⅱ（TS-NSGA-Ⅱ）算法实现任务调度。实验结果表明，与原有的负载均衡方案相比，该方法平均每次任务执行时间节省约47-55 s。同时，所有预定减速器之间的硬盘占用差异为1.254千分之一，比原方案提高了26.6%。版权所有（C）2016约翰威利父子有限公司
A heterogeneous cloud system, for example, a Hadoop 2.6.0 platform, provides distributed but cohesive services with rich features on large-scale management, reliability, and error tolerance. As big data processing is concerned, newly built cloud clusters meet the challenges of performance optimization focusing on faster task execution and more efficient usage of computing resources. Presently proposed approaches concentrate on temporal improvement, that is, shortening MapReduce time, but seldom focus on storage occupation; however, unbalanced cloud storage strategies could exhaust those nodes with heavy MapReduce cycles and further challenge the security and stability of the entire cluster. In this paper, an adaptive method is presented aiming at spatial-temporal efficiency in a heterogeneous cloud environment. A prediction model based on an optimized Kernel-based Extreme Learning Machine algorithm is proposed for faster forecast of job execution duration and space occupation, which consequently facilitates the process of task scheduling through a multi-objective algorithm called time and space optimized NSGA-II (TS-NSGA-II). Experiment results have shown that compared with the original load-balancing scheme, our approach can save approximate 47-55 s averagely on each task execution. Simultaneously, 1.254 parts per thousand of differences on hard disk occupation were made among all scheduled reducers, which achieves 26.6% improvement over the original scheme. Copyright (C) 2016 John Wiley & Sons, Ltd.