Eager: Collaborative Research: DiRecMR: Reconciling the Dichotomy of MapReduce for Efficient Speculation and Resilience

MapReduce系统具有处理海量数据的强大能力，已成为政府、学术和行业组织的研究目标。然而，他们的任务管理和故障处理策略没有认识到存在于其固有的两个阶段(映射和减少)之间的默契的二分法。这导致了许多关键问题，如资源利用不足、任务执行时间延长、目光短浅的推测和失败放大。该项目采用了理论分析、模拟和建模以及系统设计和实现方法的变革性组合，以协调MapReduce二分法。该项目的技术对部署MapReduce系统并支持大数据应用的所有组织都有潜在的影响，包括商业分析、社交网络和科学计算研究。该项目没有对系统行为进行经验分析来准确定位资源管理和任务调度异常，而是从不同的角度看待MapReduce的效率和弹性，并为Hadoop MapReducer容器的迁移制定了马尔科夫链，并为Map和Reduce任务的排队建立了分叉-连接模型。这些公式有助于对MapReduce的二分法进行理论分析，并有助于阐明其对大规模工作负载的渐近行为的影响。该项目旨在将仿真和真实系统开发融合在一起，解决了二分法带来的短视推测，解放了任务推测的范围，确保了任务的弹性而不会放大失败。开发这些技术是为了增强诸如YAR和Spark之类的MapReduce平台。除了MapReduce系统的目标之外，该项目的研究还解决了分布式分析环境中的一个一般问题。

MapReduce systems have great capabilities in processing large amounts of data and have become a research target for governmental, academic and industrial organizations. However, their task management and fault handling policies do not recognize a tacit dichotomy that exists between its inherent two phases (map and reduce). This results in a number of critical issues, such as resource underutilization, prolonged task execution, myopic speculation, and failure amplifications. This project adopts a transformative combination of theoretical analysis, simulation and modeling, and systems design and implementation approaches in order to reconcile the dichotomy of MapReduce. The techniques from this project are potentially impactful to all organizations that deploy MapReduce systems and support Big Data applications from business analytics, social networks, and scientific computing research.Instead of empirical analysis of system behaviors to pinpoint resource management and task scheduling abnormalities, this project takes a different perspective on MapReduce efficiency and resilience, and formulates a Markov chain for the transition of Hadoop MapReduce containers, and a fork-join model for the queueing of map and reduce tasks. These formulations facilitate a theoretical analysis of the dichotomy of MapReduce and help shed light on its impact to asymptotic behaviors of large-scale workloads. This project aims to blend simulation and real system development together, and addresses the myopic speculation caused by dichotomy, liberates the scope of task speculation, and ensures task resilience without failure amplifications. These techniques are developed to enhance MapReduce platforms such as YARN and Spark. Besides the target on MapReduce systems, the research from this project addresses a general issue in distributed analytics environments.