Eager: Collaborative Research: DiRecMR: Reconciling the Dichotomy of MapReduce for Efficient Speculation and Resilience

Eager：协作研究：DiRecMR：调和 MapReduce 的二分法以实现高效推测和弹性

• 批准号: 1744317
• 负责人: Xian-He Sun
• 金额: $ 8万
• 依托单位: Illinois Institute of Technology
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-08-01 至 2018-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1744317&HistoricalAwards=false
• 关键词: Eager; Collaborative; Research; DiRecMR; Reconciling

MapReduce systems have great capabilities in processing large amounts of data and have become a research target for governmental, academic and industrial organizations. However, their task management and fault handling policies do not recognize a tacit dichotomy that exists between its inherent two phases (map and reduce). This results in a number of critical issues, such as resource underutilization, prolonged task execution, myopic speculation, and failure amplifications. This project adopts a transformative combination of theoretical analysis, simulation and modeling, and systems design and implementation approaches in order to reconcile the dichotomy of MapReduce. The techniques from this project are potentially impactful to all organizations that deploy MapReduce systems and support Big Data applications from business analytics, social networks, and scientific computing research.Instead of empirical analysis of system behaviors to pinpoint resource management and task scheduling abnormalities, this project takes a different perspective on MapReduce efficiency and resilience, and formulates a Markov chain for the transition of Hadoop MapReduce containers, and a fork-join model for the queueing of map and reduce tasks. These formulations facilitate a theoretical analysis of the dichotomy of MapReduce and help shed light on its impact to asymptotic behaviors of large-scale workloads. This project aims to blend simulation and real system development together, and addresses the myopic speculation caused by dichotomy, liberates the scope of task speculation, and ensures task resilience without failure amplifications. These techniques are developed to enhance MapReduce platforms such as YARN and Spark. Besides the target on MapReduce systems, the research from this project addresses a general issue in distributed analytics environments.

MapReduce系统具有处理大量数据的强大能力，已成为政府、学术和工业组织的研究目标。然而，他们的任务管理和故障处理策略并没有认识到其固有的两个阶段（map和reduce）之间存在的隐性二分法。 这导致了一些关键问题，如资源利用不足，延长任务执行，短视的投机，和失败放大。该项目采用了理论分析，模拟和建模，系统设计和实现方法的变革性组合，以调和MapReduce的二分法。该项目的技术对所有部署MapReduce系统并支持商业分析、社交网络和科学计算研究等大数据应用的组织都具有潜在的影响力。该项目从不同的角度看待MapReduce的效率和弹性，而不是通过实证分析系统行为来查明资源管理和任务调度异常，提出了Hadoop MapReduce容器迁移的马尔可夫链模型和映射与归约任务调度的fork-join模型。这些公式有助于对MapReduce的二分法进行理论分析，并有助于揭示其对大规模工作负载的渐近行为的影响。该项目旨在将仿真和真实的系统开发融合在一起，解决二分法导致的短视推测，解放任务推测的范围，并确保任务弹性而不放大故障。这些技术的开发是为了增强MapReduce平台，如YARN和Spark。除了MapReduce系统的目标之外，该项目的研究还解决了分布式分析环境中的一个普遍问题。

项目成果

SciDP: Support HPC and Big Data Applications via Integrated Scientific Data Processing

• DOI: 10.1109/cluster.2018.00023
• 发表时间: 2018-09
• 期刊: 2018 IEEE International Conference on Cluster Computing (CLUSTER)
• 作者: Kun Feng;Xian-He Sun;Xi Yang;Shujia Zhou

Vidya: Performing Code-Block I/O Characterization for Data Access Optimization

• DOI: 10.1109/hipc.2018.00036
• 发表时间: 2018-12
• 期刊: 2018 IEEE 25th International Conference on High Performance Computing (HiPC)
• 作者: H. Devarajan;Anthony Kougkas;Prajwal Challa;Xian-He Sun