Execution Modeling and Analytics for Large-scale and Data-intensive Software Applications

软件应用程序产生日志来保存执行记录。分析这些日志是为了检测和跟踪任何事件、错误、故障或异常。新兴的软件应用程序正在变得大规模、复杂和数据密集。这类应用程序被广泛使用，它们产生的日志数据量大、速度快、种类多。由于缺乏关于记录什么以及如何记录的标准、指导方针和最佳实践，大规模地管理和分析日志变得具有挑战性。目前可用的解决方案依赖于有限的信息，并且大多局限于以自然语言描述的半结构化文本文件的形式生成日志。将这些日志作为文本文件进行处理，包括手动扫描，或使用执行基本爬行和多次遍历的工具。这使得软件应用程序的监控和管理具有挑战性。主要的限制是：(1)缺乏正式的日志数据建模，(2)缺乏结构化日志数据的指导方针，(3)缺乏在大数据规模下处理软件应用程序生成的日志的分析技术。***本研究项目旨在为软件应用程序，特别是大规模、复杂和数据密集型应用程序，发现和实现日志数据建模和分析方面的进步。我们将通过构建基于数据建模和大数据分析的集成解决方案来应对这些挑战，以高效地收集、组织和有效地分析日志。首先，将构建数据建模技术，使日志具有高度结构化、良好的表达和机器可读性。它将作为构建软件应用程序登录标准的基础。其次，将建立专门的、可扩展的、有效的定量和定性分析技术，并与数据建模技术相结合，对大规模数据密集型软件应用程序产生的日志进行管理和分析，同时最大限度地利用结构化、正式建模和高表达性的日志数据。长期目标是为日志构建一个集成的大数据建模和分析框架。它将(1)包括对日志的数据建模和分析，以及(2)作为开发人员、系统管理员和其他相关人员的信息系统，以促进自动分析，以及(3)帮助建立标准和最佳实践，以有效地监视和管理软件应用程序。***这是一项及时的研究，将显著改善下一代大规模和数据密集型软件应用的监控。研究的影响和有用性将通过将其应用于现实生活中来证明。这一研究项目将对培养高素质人才（HQPs）做出重大贡献。这将有利于加拿大在信息管理和数据科学领域获得领先地位，这些领域对这类技能的需求正在迅速增加。

Software applications produce logs to keep records of execution. Such logs are analyzed to detect, track any events, errors, faults, or exceptions. Emerging software applications are becoming large-scale, complex, and data-intensive. Such applications are extensively utilized and produce logs as data with massive size, high speed, and wide variety. Managing and analyzing logs at a large-scale becomes challenging due to lack of standards, guidelines, best practices on what and how to log. Currently available solutions rely on limited information and are mostly confined to producing logs as semi-structured text files with descriptions in natural language. Processing of such logs as text files include manual scanning, or usage of tools that carry out basic crawling and multiple traversals. This makes monitoring and management of software applications to be challenging. Key limitations are (1) lack of formal data modeling for logs, (2) lack of guidelines for structuring log data, (3) lack of analytical techniques to process logs, produced by software applications, at a scale of big data. ***This research program aims to discover and carry out advancements in data modeling and analytics of logs for software applications, especially that are large-scale, complex, and data-intensive. We will approach the challenges by building integrated solutions based on data modeling and big data analytics to efficiently collect, organize, and effectively analyze logs. Firstly, data modeling techniques will be built to make logs highly structured, well-expressed, and machine-readable. It will serve as a basis for building standards for logging in software applications. Secondly, specialized, scalable, and effective quantitative and qualitative analytical techniques, integrated with data modeling techniques, will be built to manage, and perform analytics on logs produced by large-scale and data-intensive software applications, while maximally utilizing structured, formally modeled and high expressivity of log data. Long-term goals are to build an integrated big data modeling and analytics framework for logs. It will (1) include data modeling and analytics on logs, and (2) act as an information system for developers, system administrators, and other related staff to facilitate automated analysis, and (3) help in setting up standards and best practices to effectively monitor and manage software applications. ***This is a timely research which will significantly improve monitoring of next-generation large-scale and data-intensive software applications. Impact and usefulness of the research will be demonstrated by applying it to real-life applications. This research program will contribute significantly to the training of highly qualified personnel (HQPs). It will benefit Canada in gaining a leading position in information management and data science where demand such skills is rapidly increasing.