Performance-driven task co-scheduling for MapReduce environments
Performance-driven task co-scheduling for MapReduce environments
复制标题
DOI：
10.1109/noms.2010.5488494
复制
发表时间：
2010-04
期刊：
影响因子：
--
通讯作者：
Jordà Polo;David Carrera;Y. Becerra;M. Steinder;Ian Whalley
中科院分区：
文献类型：
--
作者：
Jordà Polo;David Carrera;Y. Becerra;M. Steinder;Ian Whalley
MapReduce是Google在2004年提出的数据驱动编程模型，特别适合于分布式数据分析应用。我们考虑在多个应用程序共享相同物理资源的环境中管理MapReduceTM应用程序。这种共享符合数据中心管理的最新趋势，这些趋势旨在整合工作负载，以实现成本和能源节约。在共享环境中，有必要根据为工作负载定义的一组性能目标来预测和管理工作负载的性能。在本文中，我们通过为MapReduce框架引入一个新的任务调度器来解决这个问题，该调度器允许对MapReduce任务进行性能驱动的管理。所提出的任务调度器动态预测并发MapReduce作业的性能并调整作业的资源分配。它允许应用程序在不过度配置物理资源的情况下实现其性能目标。
MapReduce is a data-driven programming model proposed by Google in 2004 which is especially well suited for distributed data analytics applications. We consider the management of MapReduce applications in an environment where multiple applications share the same physical resources. Such sharing is in line with recent trends in data center management which aim to consolidate workloads in order to achieve cost and energy savings. In a shared environment, it is necessary to predict and manage the performance of workloads given a set of performance goals defined for them. In this paper, we address this problem by introducing a new task scheduler for a MapReduce framework that allows performance-driven management of MapReduce tasks. The proposed task scheduler dynamically predicts the performance of concurrent MapReduce jobs and adjusts the resource allocation for the jobs. It allows applications to meet their performance objectives without over-provisioning of physical resources.