Leveraging System Behaviour Data to Improve the Process of Load Testing Large Scale Software Systems

点击翻译按钮获取中文摘要

Many large scale software systems ranging from e-commerce websites (e.g., Amazon and Ebay) to telecommunication infrastructures (e.g., BlackBerry) must support concurrent access to thousands or millions of users. Studies show that many field problems of these systems are due to their inability to scale to meet user demands, rather than feature bugs. The inability to scale causes catastrophic failures and unfavorable media coverage (e.g., the botched launch of Apple's MobileMe). To ensure the quality of these systems, load testing is a required testing procedure in addition to conventional functional testing procedures (e.g., unit testing and integration testing). Load testing is gaining more importance, as an increasing number of services (e.g., Apple's iCloud and Google Drive) are being offered in the cloud to millions or even billions of users.**Load testing, in general, refers to the practice of assessing the system behavior under load. A typical load test uses one or more load generators that simultaneously send requests to the system under test. During the course of a load test, the system under test is monitored and gigabytes or terabytes of system behaviour data (e.g., performance counters and execution logs) is recorded. The system behaviour data, which is widely available in large scale systems to support problem diagnosis and remote issue resolution, contains rich information about the external environment (e.g., network latency or the rate of the requests) as well as the internal execution state of the system (e.g., request failures or the size of the request queues). However, due to the size and complexity of such data, load testing practitioners currently use it in a limited manner: mainly for ad-hoc high level manual checks (e.g., crash checks and memory leak checks). Little software testing research has been done to improve the load testing process with such valuable information. **The long term goal of this research is to leverage the rich information contained in the system behaviour data to improve the process of load testing large scale software systems. This proposed research aims to improve the theory and practices in all three phases of a load test: test design, test execution and test analysis. The following short term research objectives are proposed towards the long term goal: (1) Systematically Validating Load Test Suites (1 PhD student); (2) Adaptive Load Test Execution (2 Master's students); (3) In-depth and Scalable Load Test Analysis (2 PhD students). The expected research outcome will be useful for load testing practitioners and software engineering researchers with interest in monitoring, testing and analyzing large scale software systems.