Evaluating Scalability Bottlenecks by Workload Extrapolation
Evaluating Scalability Bottlenecks by Workload Extrapolation
复制标题
DOI：
10.1109/mascots.2018.00039
复制
发表时间：
2018-09
期刊：
影响因子：
--
通讯作者：
Rong Shi;Yifan Gan;Yang Wang
中科院分区：
文献类型：
--
作者：
Rong Shi;Yifan Gan;Yang Wang
测试可伸缩性瓶颈需要一个大型系统来生成足够的负载，这通常是研究人员无法访问的。为了解决这个问题，本文将工作负载外推到一个瓶颈节点。激发我们的方法的关键观察是，大规模的系统通常会在小规模上重复其行为，通过运行更多次作业，运行更多相同类型的节点，或运行更多相同循环的迭代。根据这一观察，我们记录了一个节点的工作负载在小规模和推断这样的工作负载在一个大规模。为了实现这一目标，我们开发了PatternMiner，这是一种半自动工具，用于识别工作负载模式如何随规模而变化。我们已经在HDFS NameNode和YARN的资源管理器上测试了我们的方法。我们的评估表明，PatternMiner能够预测NameNode的98%的工作负载和资源管理器的83%的工作负载。此外，通过利用外推的工作负载，我们能够模拟多达60，000个节点的集群，只有8台物理机器来评估NameNode和Resource Manager。
Testing a scalability bottleneck requires a large system to generate sufficient load, which is usually not accessible to researchers. To address this problem, this paper extrapolates the workload to a bottleneck node. The key observation that motivates our approach is that systems at a large scale are often repeating their behaviors at small scales, by running a job more times, running more nodes of the same type, or running more iterations of the same loop. Following this observation, we record a node's workloads at small scales and extrapolate such workload at a large scale. Towards this goal, we have developed PatternMiner, a semi-automatic tool to identify how workload patterns change with scale. We have tested our method on HDFS NameNode and YARN's Resource Manager. Our evaluation shows that PatternMiner is able to predict 98% of the workloads for NameNode and 83% of the workloads for the Resource Manager. Furthermore, by utilizing the extrapolated workload, we are able to emulate a cluster of up to 60,000 nodes with only 8 physical machines to evaluate NameNode and Resource Manager.