Mitigating Risks Associated with Big Data Solutions

今天无处不在的通信和计算技术产生的数据量远远超过十年前的数据量。大数据技术和服务市场预计将以27%的复合年增长率从2012年的98亿美元增长到2017年的324亿美元，导致产生的技术数据量增加数倍。现代组织有前所未有的机会获得有关其产品，流程和服务的新知识，这些知识可用于提高项目效率，改进产品和服务，识别问题，采取先发制人的措施，并使业务目标适应新的机会。新知识的应用包括预测模型，例如减少医院再入院率、交通拥堵和不必要的发电。***利用这些机会需要解决处理大数据（BDS）的解决方案带来的挑战。这些复杂的解决方案具有许多动态组件，例如分布式计算节点、网络、数据库、中间件和业务智能层。任何组件与其他组件的交互都可能失败，可能导致解决方案的崩溃失败或质量下降（例如，性能、可靠性、安全性）。因此，它们的发展和维护都具有挑战性。***大数据环境缺乏支持北斗系统有序发展和维护的方法、工具、流程和技术。尽管在处理北斗系统产生的数据方面有大量知识，但大多数技术无法处理北斗系统产生的大量操作数据。我的目标是通过实现以下目标来帮助改进北斗系统的测试和维护：1)建立北斗系统的缺陷预测模型，以改进北斗系统的一般测试和维护流程；2)对BDS进行跟踪分析，加快根本原因的确定和冗余测试用例的移除。为了达到目标，我将创建新的可扩展的方法、技术和工具来处理操作数据。据我所知，这些目标是新颖的，对实践很有意义。预期结果将有助于建立北斗系统自动问题确定技术的初步理论。将结果转移到加拿大工业将提高产品质量，加速缺陷检测和修复，允许开发人员创建更多新功能，并减少维护工作和投资。********

Today's ubiquitous communication and computing technologies generate magnitudes of data far beyond that available even a decade ago. The Big Data technology and services market is expected to grow at a compound annual rate of 27% from $9.8 billion in 2012 to $32.4 billion in 2017, leading to multi-fold increase in the amount of technical data generated. Modern organisations have unprecedented opportunities to gain new knowledge about their products, processes and services, which can be used to make projects more efficient, improve products and services, identify problems, take pre-emptive measures, and adapt business goals to new opportunities. Applications of new knowledge include predictive models which reduce, for example, hospital readmission rates, traffic congestion, and unnecessary power generation.*** Exploiting these opportunities requires addressing challenges posed by solutions for processing Big Data (BDS). These complex solutions have many dynamic components, such as distributed compute nodes, networks, databases, middleware, and business intelligence layers. Any component can fail its interactions with others, possibly leading to crashing failure of the solution or quality degradation (e.g., performance, reliability, security). Therefore, they are challenging to develop and maintain.*** Big Data environments lack methods, tools, processes and techniques to support the disciplined development and maintenance of BDS. Despite a significant body of knowledge on processing data generated by BDS, most techniques cannot process the volumes of operational data generated by BDS. My goal is to help improve testing and maintenance of the BDS by reaching the following objectives: 1) Build defect prediction models for the BDS to improve its General Testing and Maintenance process; 2) Develop trace analysis for the BDS to speed up root cause determination and the removal of redundant test cases. To reach the objectives, I will create novel scalable methods, techniques, and tools capable of processing the operational data.*** To the best of my knowledge, these objectives are novel and significant for practice. The anticipated results will help create a preliminary theory of automated problem determination techniques for BDS. Transfer of the results to Canadian industry will improve product quality and speed up defect detection and fixing, allowing developers to create more new functionality, and reduce maintenance effort and investment.********