CSR: Small: Moving MapReduce into the Cloud: Flexibility, Efficiency, and Elasticity

MapReduce是一种基于商用硬件集群的并行和分布式编程模型，已经成为处理大型数据集的事实上的标准。虽然MapReduce为并行编程提供了一个简单而通用的接口，但它在云中运行时会遇到一些问题，包括集群资源利用率低、可伸缩性不佳以及对多租户的支持不佳。该项目探索和设计了新的技术，让MapReducte充分利用云中灵活和弹性的资源分配的好处，同时解决服务器虚拟化带来的开销和问题。它允许以灵活且经济高效的方式执行大数据分析，从而扩大了影响。该项目还涉及行业协作、课程开发，并提供了更多途径将女性、少数族裔和代表性不足的学生带入研究和研究生课程。在云中运行MapReduce具有许多好处，包括快速部署、高可用性、按需弹性和安全的多租户。然而，简单地将MapReduce迁移到云环境并不能充分利用这些好处。MapReduce运行时与云资源管理之间的语义鸿沟，以及云管理程序中MapReduce工作负载的缺乏优化，使得实现灵活性、效率和弹性变得困难。该项目开发了一种协调MapReduce和云的协同方法。这项研究集中在两个关键设计上：1)准虚拟的MapReduce，它是MapReduce的一个增强，可以主动地使作业执行适应云动态，包括干扰和硬件异构性；2)MapReducCloud，它是对MapReduce感知的云资源分配和调度的优化集合。该项目将计算机系统实验与严谨的系统设计相结合，提高了MapReduce在云中的灵活性、效率和弹性。强调了MapReduce在异构云环境和动态云环境中的适应性，提出了跨层优化来释放云系统的潜力，并确保了对MapReduce工作负载的优化不会损害高资源利用率和多租户公平性的要求。

MapReduce, a parallel and distributed programming model on clusters of commodity hardware, has emerged as the de facto standard for processing large data sets. Although MapReduce provides a simple and generic interface for parallel programming, it incurs several problems when running in the cloud including low cluster resource utilization, suboptimal scalability and poor multi-tenancy support. This project explores and designs new techniques that let MapReduce fully exploit the benefits of flexible and elastic resource allocations in the cloud while addressing the overhead and issues caused by server virtualization. It broadens impact by allowing a flexible and cost-effective way to perform big data analytics. This project also involves industry collaboration, curriculum development, and provides more avenues to bring women, minority, and underrepresented students into research and graduate programs.Running MapReduce in the cloud offers many benefits, including rapid deployment, high availability, on-demand elasticity and secure multi-tenancy. However, a simple migration of MapReduce to the cloud environment does not fully exploit these benefits. The semantic gap between MapReduce runtime and cloud resource management, and the lack of optimizations of MapReduce workloads in cloud hypervisors, together make it difficult to attain flexibility, efficiency and elasticity. This project develops a synergistic approach for coordinating MapReduce and the cloud. This research centers on two key designs: 1) para-virtualized MapReduce, an enhancement of MapReduce to actively adapt job execution to the cloud dynamics, including interference and hardware heterogeneity; 2) MapReduce cloud, a collection of optimizations for MapReduce-aware cloud resource allocation and scheduling. This project combines computer system experimentations with rigorous system design to improve the flexibility, efficiency and elasticity of MapReduce in the cloud. It emphasizes the adaptability of MapReduce in a heterogeneous and dynamic cloud environment, proposes cross-layer optimizations to unlock the potential of cloud systems, and ensures that optimizations for MapReduce workloads do not compromise the requirements for high resource utilization and multi-tenant fairness.