CSR: Small: Moving MapReduce into the Cloud: Flexibility, Efficiency, and Elasticity

CSR：小：将 MapReduce 移至云端：灵活性、效率和弹性

• 批准号: 1422119
• 负责人: Xiaobo Zhou
• 金额: $ 40万
• 依托单位: University of Colorado at Colorado Springs
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2014
• 资助国家: 美国
• 起止时间: 2014-10-01 至 2019-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1422119&HistoricalAwards=false
• 关键词: CSR; Small; Moving; MapReduce; into

MapReduce, a parallel and distributed programming model on clusters of commodity hardware, has emerged as the de facto standard for processing large data sets. Although MapReduce provides a simple and generic interface for parallel programming, it incurs several problems when running in the cloud including low cluster resource utilization, suboptimal scalability and poor multi-tenancy support. This project explores and designs new techniques that let MapReduce fully exploit the benefits of flexible and elastic resource allocations in the cloud while addressing the overhead and issues caused by server virtualization. It broadens impact by allowing a flexible and cost-effective way to perform big data analytics. This project also involves industry collaboration, curriculum development, and provides more avenues to bring women, minority, and underrepresented students into research and graduate programs.Running MapReduce in the cloud offers many benefits, including rapid deployment, high availability, on-demand elasticity and secure multi-tenancy. However, a simple migration of MapReduce to the cloud environment does not fully exploit these benefits. The semantic gap between MapReduce runtime and cloud resource management, and the lack of optimizations of MapReduce workloads in cloud hypervisors, together make it difficult to attain flexibility, efficiency and elasticity. This project develops a synergistic approach for coordinating MapReduce and the cloud. This research centers on two key designs: 1) para-virtualized MapReduce, an enhancement of MapReduce to actively adapt job execution to the cloud dynamics, including interference and hardware heterogeneity; 2) MapReduce cloud, a collection of optimizations for MapReduce-aware cloud resource allocation and scheduling. This project combines computer system experimentations with rigorous system design to improve the flexibility, efficiency and elasticity of MapReduce in the cloud. It emphasizes the adaptability of MapReduce in a heterogeneous and dynamic cloud environment, proposes cross-layer optimizations to unlock the potential of cloud systems, and ensures that optimizations for MapReduce workloads do not compromise the requirements for high resource utilization and multi-tenant fairness.

MapReduce是一种基于商用硬件集群的并行和分布式编程模型，已成为处理大型数据集的事实标准。虽然MapReduce为并行编程提供了一个简单而通用的接口，但它在云中运行时会遇到一些问题，包括集群资源利用率低，可扩展性不佳和多租户支持差。这个项目探索和设计了新的技术，让MapReduce充分利用云中灵活和弹性资源分配的好处，同时解决服务器虚拟化带来的开销和问题。它通过允许以灵活且具有成本效益的方式执行大数据分析来扩大影响。该项目还涉及行业合作、课程开发，并为女性、少数民族和代表性不足的学生提供更多途径，让他们进入研究和研究生课程。在云中运行MapReduce提供了许多好处，包括快速部署、高可用性、按需弹性和安全多租户。然而，简单地将MapReduce迁移到云环境并不能充分利用这些好处。MapReduce运行时和云资源管理之间的语义鸿沟，以及云管理程序中MapReduce工作负载的优化不足，共同导致难以实现灵活性，效率和弹性。该项目开发了一种协调MapReduce和云的协同方法。该研究集中在两个关键设计上：1）半虚拟化MapReduce，MapReduce的增强，以主动适应云动态的作业执行，包括干扰和硬件异构性; 2）MapReduce云，MapReduce感知云资源分配和调度的优化集合。该项目将计算机系统实验与严格的系统设计相结合，以提高MapReduce在云中的灵活性，效率和弹性。它强调MapReduce在异构和动态云环境中的适应性，提出跨层优化以释放云系统的潜力，并确保MapReduce工作负载的优化不会损害高资源利用率和多租户公平性的要求。