异构并行环境下的MapReduce资源调度模型与方法研究

如何构建面向CPU/GPU异构环境的Map-Reduce系统将有助于解决超级计算在大数据处理方面的实际应用难题。本课题首先通过分析异构机群系统的特点，提出其面向Map-Reduce的结构划分策略，并由此提出 Map-Reduce在大数据处理中的随机任务图模型，以描述随机任务间的基本关系。其次提出了基于随机概率分布的任务量动态预测算法，并在此基础上实现了基于计算任务类型的动态资源调度模型。最后通过在CPU/GPU节点上建立面向大数据处理的线程级负载均衡及节点容错模型，推进了Map-Reduce资源调度模型在异构机群上的实现。本课题将基于国家超算长沙中心的天河1号异构超级计算机系统对本项目的理论和原型进行全面的性能评估与测试。本项目中的所建立的异构环境下的Map-Reduce资源调度模型将是云服务在超算平台上部署和应用的突破，是提升大数据处理并行计算性能的关键，具有重要的理论与实际价值。

Developing CPU/GPU based heterogeneous Map-Reduce systems helps to solve Big Data processing problems in super computing environments. This project first presents a Map-Reduce based heterogeneous cluster system multi-layer architecture partitioning mechanism along with a Map-Reduce scheduling algorithm for time-constrain heterogeneous systems, and establishes a random task graph models for Map-Reduce in processing big data. Secondly, this project develops a random-probability distribution based dynamic workload prediction model. A dynamic resource management model is then built on the basis of the prediction model. Finally, this project develops a thread-level load-balancing and fault-tolerant model for Big Data processing on CPU/GPU nodes and achieves a Map-Reduce resource scheduling model on heterogeneous cluster systems. The project is thoroughly tested and evaluated on the Tianhe-1A heterogeneous supercomputing system at the National Supercomputing (Changsha) Center. The Map-Reduce resource management model that is built from this project will be a breakthrough in applying could services on supercomputing platforms. Furthermore, the research findings from this project offers theoretical foundation and experimental reference to Map-Reduce heterogeneous systems designs.