III: Small: MapReduce Workload Management

欺诈检测、基因组测序和数据中心管理等不同领域的研究人员和决策者每天需要处理数TB的数据。许多领域正在转向MapReduce系统来处理这样不断增长的数据集。因此，相对年轻的MapReduce生态系统必须支持复杂的工作负载，包括用于报告生成的声明性查询、用于机器学习任务的MapReduce程序和大型作业工作流。此外，弹性和即用即付的云平台为MapReduce工作负载管理带来了新的挑战和机遇。该项目正在构建Hadoop AutoAdmin系统，以实现MapReduce工作负载管理的自动化。据PI所知，Hadoop AutoAdmin是第一个解决这个具有挑战性的问题的系统，随着越来越多的用户采用MapReduce，这个问题将变得越来越重要。Hadoop AutoAdmin有三个研究目标。第一个重点是基于一项涉及来自多个应用程序域的工作负载和数据以及云上不同集群配置的全面经验研究，了解和描述MapReduce工作负载的行为。第二个目标是开发一个易于使用且高效的仓库，用于存储、检索和可视化各种形式的工作负载监控数据。来自这些活动的模型和见解将推动开发用于工作负载管理的端到端算法的第三个推动力。该项目可能会在数据泛滥的国家重要领域(如安全和医疗保健)产生重大影响。Hadoop AutoAdmin将提高云平台的工作效率、系统利用率和成本效益。技术贡献将被广泛传播，该系统将公开发布。

Researchers and decision makers in diverse fields such as fraud detection, genome sequencing, and datacentermanagement need to process many terabytes of data every day. Many fields are turning to MapReduce systems to process such growing datasets. Consequently, the relatively young MapReduceecosystem has to support complex workloads that include declarative queries for report generation, MapReduce programs for machine learning tasks, and large job workflows.Furthermore, elastic and pay-as-you-go cloud platforms pose novel challenges and opportunities for MapReduce workload management.This project is building the Hadoop AutoAdmin system for automating MapReduce workload management. To the PI's knowledge, Hadoop AutoAdmin is the first system to address this challenging problem that will become increasingly important as a broad class of users adopt MapReduce. Hadoop AutoAdmin has three research thrusts. The first thrust is to understand and characterize the behavior of MapReduce workloads based on a comprehensive empirical study involving workloads and data from multiple application domains as well as different cluster configurations on the cloud. The second thrust is to develop an easy-to-use and efficient warehouse to store, retrieve, and visualize the diverse forms of workload monitoring data. The models and insights from these activities will drive the third thrust of developing end-to-end algorithms for workload management.This project can have significant impact in areas of national importance like security and healthcare that are inundated with data. Hadoop AutoAdmin will improve worker productivity, system utilization, and cost-effectiveness of cloud platforms. The technical contributions will be disseminated broadly and the system released publicly.