Mitigating Risks Associated with Big Data Solutions

如今无处不在的通信和计算技术产生的数据量远远超过十年前的水平。大数据技术和服务市场预计将以27%的复合年增长率从2012年的98亿美元增长到2017年的324亿美元，从而导致产生的技术数据量增加数倍。现代组织拥有前所未有的机会来获取有关其产品、流程和服务的新知识，这些知识可用于提高项目效率、改进产品和服务、发现问题、采取先发制人的措施并使业务目标适应新的机遇。新知识的应用包括预测模型，它可以减少例如医院再住院率、交通拥堵和不必要的发电。 利用这些机遇需要应对大数据(BDS)处理解决方案带来的挑战。这些复杂的解决方案具有许多动态组件，如分布式计算节点、网络、数据库、中间件和商业智能层。任何组件都可能导致其与其他组件的交互失败，可能导致解决方案崩溃或质量下降(例如，性能、可靠性、安全性)。因此，它们的开发和维护是具有挑战性的。 大数据环境缺乏方法、工具、流程和技术来支持规范的BDS开发和维护。尽管有大量关于处理BDS生成的数据的知识，但大多数技术无法处理BDS生成的大量业务数据。我的目标是通过实现以下目标来帮助改进BDS的测试和维护：1)为BDS构建缺陷预测模型，以改进其一般测试和维护流程；2)为BDS开发跟踪分析，以加快根本原因确定和删除多余的测试用例。为了实现这些目标，我将创建能够处理运营数据的新的可扩展方法、技术和工具。 据我所知，这些目标是新颖的，对实践具有重要意义。预期的结果将有助于为BDS创建自动问题确定技术的初步理论。将结果转移到加拿大行业将提高产品质量，加快缺陷检测和修复，使开发人员能够创建更多新功能，并减少维护工作和投资。

Today’s ubiquitous communication and computing technologies generate magnitudes of data far beyond that available even a decade ago. The Big Data technology and services market is expected to grow at a compound annual rate of 27% from $9.8 billion in 2012 to $32.4 billion in 2017, leading to multi-fold increase in the amount of technical data generated. Modern organisations have unprecedented opportunities to gain new knowledge about their products, processes and services, which can be used to make projects more efficient, improve products and services, identify problems, take pre-emptive measures, and adapt business goals to new opportunities. Applications of new knowledge include predictive models which reduce, for example, hospital readmission rates, traffic congestion, and unnecessary power generation. Exploiting these opportunities requires addressing challenges posed by solutions for processing Big Data (BDS). These complex solutions have many dynamic components, such as distributed compute nodes, networks, databases, middleware, and business intelligence layers. Any component can fail its interactions with others, possibly leading to crashing failure of the solution or quality degradation (e.g., performance, reliability, security). Therefore, they are challenging to develop and maintain. Big Data environments lack methods, tools, processes and techniques to support the disciplined development and maintenance of BDS. Despite a significant body of knowledge on processing data generated by BDS, most techniques cannot process the volumes of operational data generated by BDS. My goal is to help improve testing and maintenance of the BDS by reaching the following objectives: 1) Build defect prediction models for the BDS to improve its General Testing and Maintenance process; 2) Develop trace analysis for the BDS to speed up root cause determination and the removal of redundant test cases. To reach the objectives, I will create novel scalable methods, techniques, and tools capable of processing the operational data. To the best of my knowledge, these objectives are novel and significant for practice. The anticipated results will help create a preliminary theory of automated problem determination techniques for BDS. Transfer of the results to Canadian industry will improve product quality and speed up defect detection and fixing, allowing developers to create more new functionality, and reduce maintenance effort and investment.