III: Small: MapReduce Workload Management

三：小：MapReduce 工作负载管理

• 批准号: 1218981
• 负责人: Shivnath Babu
• 金额: $ 30万
• 依托单位: Duke University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2012
• 资助国家: 美国
• 起止时间: 2012-09-01 至 2016-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1218981&HistoricalAwards=false
• 关键词: III; Small; MapReduce; Workload; Management

Researchers and decision makers in diverse fields such as fraud detection, genome sequencing, and datacentermanagement need to process many terabytes of data every day. Many fields are turning to MapReduce systems to process such growing datasets. Consequently, the relatively young MapReduceecosystem has to support complex workloads that include declarative queries for report generation, MapReduce programs for machine learning tasks, and large job workflows.Furthermore, elastic and pay-as-you-go cloud platforms pose novel challenges and opportunities for MapReduce workload management.This project is building the Hadoop AutoAdmin system for automating MapReduce workload management. To the PI's knowledge, Hadoop AutoAdmin is the first system to address this challenging problem that will become increasingly important as a broad class of users adopt MapReduce. Hadoop AutoAdmin has three research thrusts. The first thrust is to understand and characterize the behavior of MapReduce workloads based on a comprehensive empirical study involving workloads and data from multiple application domains as well as different cluster configurations on the cloud. The second thrust is to develop an easy-to-use and efficient warehouse to store, retrieve, and visualize the diverse forms of workload monitoring data. The models and insights from these activities will drive the third thrust of developing end-to-end algorithms for workload management.This project can have significant impact in areas of national importance like security and healthcare that are inundated with data. Hadoop AutoAdmin will improve worker productivity, system utilization, and cost-effectiveness of cloud platforms. The technical contributions will be disseminated broadly and the system released publicly.

欺诈检测、基因组测序和制药企业管理等不同领域的研究人员和决策者每天都需要处理数TB的数据。许多领域正在转向MapReduce系统来处理这些不断增长的数据集。因此，相对年轻的MapReduce生态系统必须支持复杂的工作负载，包括用于报告生成的声明式查询，用于机器学习任务的MapReduce程序以及大型作业工作流。此外，弹性和按需付费的云平台为MapReduce工作负载管理带来了新的挑战和机遇。本项目正在构建Hadoop AutoAdmin系统，以自动化MapReduce工作负载管理。据PI所知，Hadoop AutoAdmin是第一个解决这个具有挑战性的问题的系统，随着广泛的用户采用MapReduce，这个问题将变得越来越重要。Hadoop AutoAdmin有三个研究重点。第一个重点是基于全面的实证研究来理解和描述MapReduce工作负载的行为，该研究涉及来自多个应用程序域的工作负载和数据以及云上不同的集群配置。第二个目标是开发一个易于使用且高效的仓库，以存储、检索和可视化各种形式的工作负载监控数据。这些活动的模型和见解将推动开发端到端工作负载管理算法的第三个推力。该项目可能会对国家重要领域产生重大影响，如数据泛滥的安全和医疗保健。Hadoop AutoAdmin将提高员工的工作效率、系统利用率和云平台的成本效益。技术贡献将广泛传播，系统将公开发布。