A statistics-based performance testing methodology for cloud applications
A statistics-based performance testing methodology for cloud applications
复制标题
DOI：
10.1145/3338906.3338912
复制
发表时间：
2019-08
期刊：
影响因子：
--
通讯作者：
Sen He;Glenna Manns;John Saunders;Wei Wang;L. Pollock;M. Soffa
中科院分区：
文献类型：
--
作者：
Sen He;Glenna Manns;John Saunders;Wei Wang;L. Pollock;M. Soffa
资源拥有的低成本和灵活性使得用户越来越多地将其应用程序移植到云。为了充分实现云服务的成本效益，用户通常需要可靠地了解其应用程序的执行性能。然而，由于云应用程序所经历的随机性能波动，公共云的黑匣子性质以及云使用成本，在云上进行测试以获得准确的性能结果是非常困难的。在本文中，我们提出了一种新的云性能测试方法，称为PT4Cloud。通过采用似然理论的非参数统计方法和bootstrap方法，PT4Cloud提供了可靠的停止条件，以获得具有置信带的高度准确的性能分布。这些统计方法还允许用户指定直观的准确性目标，并轻松地在准确性和测试成本之间进行权衡。我们使用Amazon Web Service和Chameleon云上的33个基准配置对PT4Cloud进行了评估。与从广泛的性能测试中获得的性能数据相比，PT4Cloud提供的测试结果平均准确率为95.4%，同时将测试运行次数减少了62%。我们还为PT4Cloud提出了两种测试执行减少技术，可以减少90.1%的测试运行次数，同时保持91%的平均准确率。我们将我们的技术与其他三种技术进行了比较，发现我们的结果更加准确。
The low cost of resource ownership and flexibility have led users to increasingly port their applications to the clouds. To fully realize the cost benefits of cloud services, users usually need to reliably know the execution performance of their applications. However, due to the random performance fluctuations experienced by cloud applications, the black box nature of public clouds and the cloud usage costs, testing on clouds to acquire accurate performance results is extremely difficult. In this paper, we present a novel cloud performance testing methodology called PT4Cloud. By employing non-parametric statistical approaches of likelihood theory and the bootstrap method, PT4Cloud provides reliable stop conditions to obtain highly accurate performance distributions with confidence bands. These statistical approaches also allow users to specify intuitive accuracy goals and easily trade between accuracy and testing cost. We evaluated PT4Cloud with 33 benchmark configurations on Amazon Web Service and Chameleon clouds. When compared with performance data obtained from extensive performance tests, PT4Cloud provides testing results with 95.4% accuracy on average while reducing the number of test runs by 62%. We also propose two test execution reduction techniques for PT4Cloud, which can reduce the number of test runs by 90.1% while retaining an average accuracy of 91%. We compared our technique to three other techniques and found that our results are much more accurate.