Mitigating Risks Associated with Big Data Solutions

点击翻译按钮获取中文摘要

Today's ubiquitous communication and computing technologies generate magnitudes of data far beyond that available even a decade ago. The Big Data technology and services market is expected to grow at a compound annual rate of 27% from $9.8 billion in 2012 to $32.4 billion in 2017, leading to multi-fold increase in the amount of technical data generated. Modern organisations have unprecedented opportunities to gain new knowledge about their products, processes and services, which can be used to make projects more efficient, improve products and services, identify problems, take pre-emptive measures, and adapt business goals to new opportunities. Applications of new knowledge include predictive models which reduce, for example, hospital readmission rates, traffic congestion, and unnecessary power generation.*** Exploiting these opportunities requires addressing challenges posed by solutions for processing Big Data (BDS). These complex solutions have many dynamic components, such as distributed compute nodes, networks, databases, middleware, and business intelligence layers. Any component can fail its interactions with others, possibly leading to crashing failure of the solution or quality degradation (e.g., performance, reliability, security). Therefore, they are challenging to develop and maintain.*** Big Data environments lack methods, tools, processes and techniques to support the disciplined development and maintenance of BDS. Despite a significant body of knowledge on processing data generated by BDS, most techniques cannot process the volumes of operational data generated by BDS. My goal is to help improve testing and maintenance of the BDS by reaching the following objectives: 1) Build defect prediction models for the BDS to improve its General Testing and Maintenance process; 2) Develop trace analysis for the BDS to speed up root cause determination and the removal of redundant test cases. To reach the objectives, I will create novel scalable methods, techniques, and tools capable of processing the operational data.*** To the best of my knowledge, these objectives are novel and significant for practice. The anticipated results will help create a preliminary theory of automated problem determination techniques for BDS. Transfer of the results to Canadian industry will improve product quality and speed up defect detection and fixing, allowing developers to create more new functionality, and reduce maintenance effort and investment.********