Execution Modeling and Analytics for Large-scale and Data-intensive Software Applications

软件应用程序生成日志来保存执行记录。分析这些日志以检测、跟踪任何事件、错误、故障或异常。新兴的软件应用程序正在变得大规模、复杂和数据密集型。这类应用被广泛使用，并以海量、高速、种类繁多的数据形式产生日志。由于缺乏关于记录什么和如何记录的标准、指导方针和最佳实践，大规模管理和分析日志变得具有挑战性。当前可用的解决方案依赖于有限的信息，并且大多限于将日志生成为具有自然语言描述的半结构化文本文件。对文本文件等日志的处理包括手动扫描，或使用执行基本爬行和多次遍历的工具。这使得对软件应用程序的监控和管理变得具有挑战性。主要限制是(1)缺乏对日志的正式数据建模，(2)缺乏构建日志数据的指导方针，(3)缺乏分析技术来处理由软件应用程序产生的大数据规模的日志。*本研究计划旨在发现和开展软件应用程序，特别是大规模、复杂和数据密集型应用程序的数据建模和日志分析方面的进展。我们将通过构建基于数据建模和大数据分析的集成解决方案来应对这些挑战，以高效地收集、组织和有效地分析日志。首先，将构建数据建模技术，使日志高度结构化、表达良好和机器可读。它将作为建立登录软件应用程序标准的基础。其次，将建立专门的、可扩展的、有效的定量和定性分析技术，与数据建模技术相结合，对大规模和数据密集型软件应用程序产生的日志进行管理和分析，同时最大限度地利用日志数据的结构化、形式化建模和高可表现性。长期目标是为日志构建一个集成的大数据建模和分析框架。它将(1)包括数据建模和日志分析，(2)作为开发人员、系统管理员和其他相关人员的信息系统，以促进自动化分析，以及(3)帮助建立标准和最佳实践，以有效地监控和管理软件应用程序。*这是一项及时的研究，将显著改善对下一代大规模和数据密集型软件应用程序的监控。这项研究的影响和用处将通过将其应用于现实生活来展示。这项研究计划将对培养高素质人才(HQP)做出重大贡献。这将有利于加拿大在信息管理和数据科学领域获得领先地位，在这些领域，对这类技能的需求正在迅速增长。

Software applications produce logs to keep records of execution. Such logs are analyzed to detect, track any events, errors, faults, or exceptions. Emerging software applications are becoming large-scale, complex, and data-intensive. Such applications are extensively utilized and produce logs as data with massive size, high speed, and wide variety. Managing and analyzing logs at a large-scale becomes challenging due to lack of standards, guidelines, best practices on what and how to log. Currently available solutions rely on limited information and are mostly confined to producing logs as semi-structured text files with descriptions in natural language. Processing of such logs as text files include manual scanning, or usage of tools that carry out basic crawling and multiple traversals. This makes monitoring and management of software applications to be challenging. Key limitations are (1) lack of formal data modeling for logs, (2) lack of guidelines for structuring log data, (3) lack of analytical techniques to process logs, produced by software applications, at a scale of big data. ***This research program aims to discover and carry out advancements in data modeling and analytics of logs for software applications, especially that are large-scale, complex, and data-intensive. We will approach the challenges by building integrated solutions based on data modeling and big data analytics to efficiently collect, organize, and effectively analyze logs. Firstly, data modeling techniques will be built to make logs highly structured, well-expressed, and machine-readable. It will serve as a basis for building standards for logging in software applications. Secondly, specialized, scalable, and effective quantitative and qualitative analytical techniques, integrated with data modeling techniques, will be built to manage, and perform analytics on logs produced by large-scale and data-intensive software applications, while maximally utilizing structured, formally modeled and high expressivity of log data. Long-term goals are to build an integrated big data modeling and analytics framework for logs. It will (1) include data modeling and analytics on logs, and (2) act as an information system for developers, system administrators, and other related staff to facilitate automated analysis, and (3) help in setting up standards and best practices to effectively monitor and manage software applications. ***This is a timely research which will significantly improve monitoring of next-generation large-scale and data-intensive software applications. Impact and usefulness of the research will be demonstrated by applying it to real-life applications. This research program will contribute significantly to the training of highly qualified personnel (HQPs). It will benefit Canada in gaining a leading position in information management and data science where demand such skills is rapidly increasing.