Execution Modeling and Analytics for Large-scale and Data-intensive Software Applications

软件应用程序生成日志来保存执行记录。分析这些日志以检测、跟踪任何事件、错误、故障或异常。新兴的软件应用程序正在变得大规模、复杂和数据密集型。这些应用程序被广泛使用，并产生大量的数据，高速，和各种各样的日志。由于缺乏关于记录什么和如何记录的标准、指导方针和最佳实践，大规模管理和分析日志变得具有挑战性。目前可用的解决方案依赖于有限的信息，并且大多局限于将日志生成为具有自然语言描述的半结构化文本文件。将此类日志作为文本文件处理包括手动扫描，或使用执行基本抓取和多次遍历的工具。这使得软件应用程序的监控和管理具有挑战性。主要的限制是（1）缺乏正式的日志数据建模，（2）缺乏结构化日志数据的指导方针，（3）缺乏分析技术来处理由软件应用程序产生的大数据规模的日志。* 该研究计划旨在发现和实施软件应用程序的数据建模和日志分析方面的进步，特别是大规模，复杂和数据密集型的应用程序。我们将通过构建基于数据建模和大数据分析的集成解决方案来应对挑战，以有效地收集，组织和有效地分析日志。首先，将建立数据建模技术，使日志高度结构化，良好的表达，和机器可读。它将作为建立软件应用程序登录标准的基础。其次，将建立与数据建模技术相结合的专业化，可扩展和有效的定量和定性分析技术，以管理和分析大型数据密集型软件应用程序产生的日志，同时最大限度地利用结构化，正式建模和高表达性的日志数据。长期目标是为日志构建集成的大数据建模和分析框架。它将（1）包括日志数据建模和分析，（2）作为开发人员，系统管理员和其他相关人员的信息系统，以促进自动化分析，（3）帮助建立标准和最佳实践，以有效地监控和管理软件应用程序。* 这是一项及时的研究，将显著改善对下一代大规模和数据密集型软件应用程序的监控。研究的影响和有用性将通过将其应用于实际生活中来证明。这项研究计划将大大有助于高素质人才（HQP）的培训。这将有利于加拿大在信息管理和数据科学方面取得领先地位，因为这些技能的需求正在迅速增加。

Software applications produce logs to keep records of execution. Such logs are analyzed to detect, track any events, errors, faults, or exceptions. Emerging software applications are becoming large-scale, complex, and data-intensive. Such applications are extensively utilized and produce logs as data with massive size, high speed, and wide variety. Managing and analyzing logs at a large-scale becomes challenging due to lack of standards, guidelines, best practices on what and how to log. Currently available solutions rely on limited information and are mostly confined to producing logs as semi-structured text files with descriptions in natural language. Processing of such logs as text files include manual scanning, or usage of tools that carry out basic crawling and multiple traversals. This makes monitoring and management of software applications to be challenging. Key limitations are (1) lack of formal data modeling for logs, (2) lack of guidelines for structuring log data, (3) lack of analytical techniques to process logs, produced by software applications, at a scale of big data. ***This research program aims to discover and carry out advancements in data modeling and analytics of logs for software applications, especially that are large-scale, complex, and data-intensive. We will approach the challenges by building integrated solutions based on data modeling and big data analytics to efficiently collect, organize, and effectively analyze logs. Firstly, data modeling techniques will be built to make logs highly structured, well-expressed, and machine-readable. It will serve as a basis for building standards for logging in software applications. Secondly, specialized, scalable, and effective quantitative and qualitative analytical techniques, integrated with data modeling techniques, will be built to manage, and perform analytics on logs produced by large-scale and data-intensive software applications, while maximally utilizing structured, formally modeled and high expressivity of log data. Long-term goals are to build an integrated big data modeling and analytics framework for logs. It will (1) include data modeling and analytics on logs, and (2) act as an information system for developers, system administrators, and other related staff to facilitate automated analysis, and (3) help in setting up standards and best practices to effectively monitor and manage software applications. ***This is a timely research which will significantly improve monitoring of next-generation large-scale and data-intensive software applications. Impact and usefulness of the research will be demonstrated by applying it to real-life applications. This research program will contribute significantly to the training of highly qualified personnel (HQPs). It will benefit Canada in gaining a leading position in information management and data science where demand such skills is rapidly increasing.