SaTC: CORE: Medium: Large-Scale Data Driven Anomaly Detection and Diagnosis from System Logs

检测计算机系统中的异常和异常行为是确保计算机系统安全可靠的关键部分。记录程序所采取的操作的系统日志是异常检测的一个很有前途的数据来源。然而，进行日志分析的现有实践和工具需要深厚的专业知识，并且在定义和解释可能的异常时需要大量的人力参与，这限制了它们的可扩展性和有效性。该项目的目标是通过开发一个名为DeepLog的框架，通过(a)推进自然语言处理技术，从各种日志文件中提取结构化信息，以支持跨不同数据源和时间的分析，从而提高基于日志的异常检测的技术水平；(b)开发新方法，对合法工作流程进行建模，并随时间记录事件序列。(c)采用机器学习方法来识别代表潜在异常的工作流程的偏差，以及(d)为系统管理员创建工具，帮助他们更有效地诊断可能的安全问题。这项工作将被整合到一个免费提供的软件包中，以使其他研究人员和实践系统管理员受益，并用于支持研究机构的课堂和基于研究的教育活动。对于日志解析，团队将采用命名实体识别方法来解析非结构化日志和结构化日志，其中结构不是由正则表达式预先定义的，例如，将其解析为日志事件类型和参数的结构化键值对。这些数据可以看作是一个多维特征空间，其内容受到底层程序执行的约束，因此反映了一个隐藏的结构，该结构定义了一组有效的、非异常的执行序列。为了帮助阐明这种隐藏的结构，该团队将开发基于长短期记忆（LSTM）的神经网络模型，该模型使用键和值元素从从已知正常系统运行中提取的数据中提取程序行为的语义有意义的子序列。一旦使用已知的训练数据开发了这些模型，它们就可以通过标记考虑系统、日志和模型当前状态下意外的新日志条目来应用于异常检测；它们还可以用来推断前面描述的底层工作流和隐藏结构。这些模型将通过在线学习方法、管理员对报告异常严重性的反馈以及生成对抗训练模型得到改进。生成对抗训练模型创建执行序列，尽管异常，但与嵌入在日志和基于lstm的模型中的隐藏结构密切相关。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Detecting unusual and anomalous behavior in computer systems is a critical part of ensuring they are secure and trustworthy. System logs, which record actions taken by programs, are a promising source of data for such anomaly detection. However, existing practices and tools for doing log analysis require deep expertise, as well as heavy human involvement in both defining and interpreting possible anomalies, which limits their scalability and effectiveness. This project's goal is to improve the state of the art around log-based anomaly detection by developing a framework called DeepLog through (a) advancing natural language processing techniques to extract structured information from a wide variety of log files to support analysis across different data sources and across time, (b) developing new methods to model legitimate workflows and log event sequences over time, (c) adapting machine learning methods to identify deviations from those workflows that represent potential anomalies, and (d) creating tools for system administrators to help them diagnose possible security issues more effectively and efficiently. The work will be integrated into a freely available software package to benefit both other researchers and practicing system administrators and used to support both classroom and research-based educational activities at the investigators' institutions.Toward log parsing, the team will adapt named entity recognition methods to parse unstructured logs as well as structured logs where the structure is not pre-defined by, e.g., regular expressions, into structured key-value pairs of log event types and parameters. This data can be seen as a multi-dimensional feature space whose contents are constrained by the execution of the underlying programs and thus reflects a hidden structure that defines the set of valid, non-anomalous execution sequences. To help articulate this hidden structure, the team will develop long-short-term-memory (LSTM)-based neural network models that use both the key and value elements to extract semantically meaningful subsequences of program behavior from data extracted from system runs known to be normal. Once these models are developed using known-good training data, they can be applied to anomaly detection by flagging for consideration new log entries that are unexpected given the current state of the system, logs, and model; they can also be used to infer the underlying workflows and hidden structures described earlier. These models will be improved through that online learning methods, administrators' feedback about the seriousness of reported anomalies, and generative adversarial training models which create execution sequences that, though anomalous, hew closely to the hidden structures embedded in the logs and the LSTM-based models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.