mrMoulder: A recommendation-based adaptive parameter tuning approach for big data processing platform
mrMoulder: A recommendation-based adaptive parameter tuning approach for big data processing platform
复制标题
mrMolder：一种基于推荐的大数据处理平台自适应参数调整方法
DOI：
10.1016/j.future.2018.05.080
复制
发表时间：
2019-04-01
影响因子：
7.5
通讯作者：
Li, Jingwei
中科院分区：
文献类型：
--
作者：
Cai, Lin;Qi, Yong;Li, Jingwei
如今，世界已经进入大数据时代。Hadoop和Spark等大数据处理平台越来越多地被许多应用采用，其中有许多参数可以调整，以提高大数据平台运营商的处理性能。然而，由于这些参数数量众多，且它们之间的关系复杂，手动调整参数非常耗时。因此，尽快自动配置参数以优化当前作业的性能是一项挑战。现有的自动调优方法在作业运行前往往需要一段时间才能得到最优配置，这会增加作业的总处理时间，降低集群的整体效率。在本文中，我们提出了一个自适应调优框架mrMoulder，以在短时间内为新作业推荐一个接近最优的配置。MrMoulder设置了可自我扩展的配置库和基于协同过滤的推荐引擎，加快了参数优化配置的进程。我们在Hadoop集群中部署了mrMoulder，实验结果表明，对于一个新的大数据应用，mrMoulder的推荐时间仅为现有自动调优方法的20%到30%，而推荐质量几乎保持不变。(C)2018爱思唯尔B.V.保留所有权利。
Nowadays the world has entered the big data era. Big data processing platforms, such as Hadoop and Spark, are increasingly adopted by many applications, in which there are numerous parameters that can be tuned to improve processing performance for big data platform operators. However, due to the large number of these parameters and the complex relationship among them, it is very time-consuming to manually tune parameters. Therefore, it is a challenge to automatically configure parameters as quickly as possible to optimize the performance of the current job. Existing auto-tuning methods often take a certain time before job runs to get the optimal configuration, which would increase the job's total processing time and reduce the overall efficiency of cluster. In this paper, we propose an adaptive tuning framework, mrMoulder, to recommend a near-optimal configuration for the new job in a short time. mrMoulder sets a self-extending configuration repository and a collaborative filtering based recommendation engine, to speed up the process of optimizing parameter configuration. We have deployed mrMoulder in a Hadoop cluster, and the experiment results have demonstrated that, for a new big data application, the recommend time of mrMoulder is only 20% to 30% of that for the existing auto-tuning methods, while the recommendation quality remains almost unchanged. (C) 2018 Elsevier B.V. All rights reserved.