Leveraging System Behaviour Data to Improve the Process of Load Testing Large Scale Software Systems

利用系统行为数据改进大型软件系统的负载测试过程

• 批准号: RGPIN-2014-06673
• 负责人: Jiang, ZhenMing
• 金额: $ 1.68万
• 依托单位: York University
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2015
• 资助国家: 加拿大
• 起止时间: 2015-01-01 至 2016-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=589152
• 关键词: Leveraging; System; Behaviour; Data; Improve

Many large scale software systems ranging from e-commerce websites (e.g., Amazon and Ebay) to telecommunication infrastructures (e.g., BlackBerry) must support concurrent access to thousands or millions of users. Studies show that many field problems of these systems are due to their inability to scale to meet user demands, rather than feature bugs. The inability to scale causes catastrophic failures and unfavorable media coverage (e.g., the botched launch of Apple’s MobileMe). To ensure the quality of these systems, load testing is a required testing procedure in addition to conventional functional testing procedures (e.g., unit testing and integration testing). Load testing is gaining more importance, as an increasing number of services (e.g., Apple’s iCloud and Google Drive) are being offered in the cloud to millions or even billions of users. Load testing, in general, refers to the practice of assessing the system behavior under load. A typical load test uses one or more load generators that simultaneously send requests to the system under test. During the course of a load test, the system under test is monitored and gigabytes or terabytes of system behaviour data (e.g., performance counters and execution logs) is recorded. The system behaviour data, which is widely available in large scale systems to support problem diagnosis and remote issue resolution, contains rich information about the external environment (e.g., network latency or the rate of the requests) as well as the internal execution state of the system (e.g., request failures or the size of the request queues). However, due to the size and complexity of such data, load testing practitioners currently use it in a limited manner: mainly for ad-hoc high level manual checks (e.g., crash checks and memory leak checks). Little software testing research has been done to improve the load testing process with such valuable information. The long term goal of this research is to leverage the rich information contained in the system behaviour data to improve the process of load testing large scale software systems. This proposed research aims to improve the theory and practices in all three phases of a load test: test design, test execution and test analysis. The following short term research objectives are proposed towards the long term goal: (1) Systematically Validating Load Test Suites (1 PhD student); (2) Adaptive Load Test Execution (2 Master's students); (3) In-depth and Scalable Load Test Analysis (2 PhD students). The expected research outcome will be useful for load testing practitioners and software engineering researchers with interest in monitoring, testing and analyzing large scale software systems.

从电子商务网站（例如，亚马逊和eBay）到电信基础架构（例如BlackBerry），许多大型软件系统必须支持对数千或数百万用户的同时访问。研究表明，这些系统的许多现场问题是由于它们无法扩展以满足用户需求而不是特征错误。无法扩展会导致灾难性的失败和不利的媒体报道（例如，苹果Mobileme的爆发发射）。为了确保这些系统的质量，除了传统的功能测试程序（例如，单位测试和集成测试）外，负载测试是必需的测试程序。随着越来越多的服务（例如，苹果的iCloud和Google Drive）在云中向数百万甚至数十亿个用户提供，负载测试变得越来越重要。 通常，负载测试是指评估负载下系统行为的做法。典型的负载测试使用一个或多个负载生成器，这些发电机只需将请求发送到正在测试的系统中即可。在负载测试过程中，监视正在测试的系统，并记录系统行为数据的千兆字节或terabytes（例如，性能计数器和执行日志）。系统行为数据在大规模系统中广泛使用，以支持问题诊断和远程问题解决，其中包含有关外部环境（例如，网络延迟或请求率）以及系统的内部执行状态（例如，请求故障或请求队列的大小）的丰富信息。但是，由于此类数据的大小和复杂性，负载测试从业者目前以有限的方式使用它：主要用于临时高级手动检查（例如，碰撞检查和内存泄漏检查）。很少进行软件测试研究以通过此类有价值的信息来改善负载测试过程。 这项研究的长期目标是利用系统行为数据中包含的丰富信息来改善负载测试大规模软件系统的过程。这项拟议的研究旨在改善负载测试的所有三个阶段的理论和实践：测试设计，测试执行和测试分析。提出了以下短期研究目标，以实现长期目标：（1）系统验证负载测试套件（1位PHD学生）； （2）自适应负载测试执行（2名硕士学生）； （3）深入且可扩展的负载测试分析（2位PHD学生）。预期的研究结果将对负载测试从业人员和软件工程研究人员有用，该研究人员有兴趣监视，测试和分析大型软件系统。