ROBOTune: High-Dimensional Configuration Tuning for Cluster-Based Data Analytics
ROBOTune: High-Dimensional Configuration Tuning for Cluster-Based Data Analytics
复制标题
DOI：
10.1145/3472456.3472518
复制
发表时间：
2021-08
期刊：
影响因子：
--
通讯作者：
Md. Muhib Khan;Weikuan Yu
中科院分区：
文献类型：
--
作者：
Md. Muhib Khan;Weikuan Yu
Spark因其在不同系统上启用高性能数据分析应用程序的能力而广受欢迎。它的多功能性是通过众多用户级和系统级选项实现的，这导致了指数级的配置空间，具有讽刺意味的是，这阻碍了数据分析的最佳性能。巨大的复杂性是由两个主要问题引起的：配置空间的高维和昂贵的黑盒配置-性能关系。在本文中，我们设计并开发了一个健壮的调优框架ROBOTune，它可以解决这两个问题并快速调优Spark应用程序，以实现高效的数据分析。具体而言，它通过基于随机森林的模型进行参数选择，以降低分析配置空间的维数。此外，ROBOTune采用贝叶斯优化来克服配置-性能关系的复杂性，并平衡探索和开发，以有效地定位全局最优或近最优配置。此外，ROBOTune通过缓存和记忆增强了Latin Hypercube Sampling，以提高样本配置生成的覆盖率和有效性。我们的评估结果表明，ROBOTune可以找到与BestConfig和Gunther等当代调优工具相似或更好的配置，同时平均将搜索成本分别提高1.59倍和1.53倍，最高可达2.27倍和1.71倍。
Spark is popular for its ability to enable high-performance data analytics applications on diverse systems. Its great versatility is achieved through numerous user- and system-level options, resulting in an exponential configuration space that, ironically, hinders data analytics’s optimal performance. The colossal complexity is caused by two main issues: the high dimensionality of configuration space and the expensive black-box configuration-performance relationship. In this paper, we design and develop a robust tuning framework called ROBOTune that can tackle both issues and tune Spark applications quickly for efficient data analytics. Specifically, it performs parameter selection through a Random Forests based model to reduce the dimensionality of analytics configuration space. In addition, ROBOTune employs Bayesian Optimization to overcome the complex nature of the configuration-performance relationship and balance exploration and exploitation to efficiently locate a globally optimal or near-optimal configuration. Furthermore, ROBOTune strengthens Latin Hypercube Sampling with caching and memoization to enhance the coverage and effectiveness in the generation of sample configurations. Our evaluation results demonstrate that ROBOTune finds similar or better performing configurations than contemporary tuning tools like BestConfig and Gunther while improving on search cost by 1.59 × and 1.53 × on average and up to 2.27 × and 1.71 × , respectively.