SHF: Small: Failure Events Modeling and Analysis for Proactive Management in Highly Dependable Systems

在大型计算机系统中，部件故障不再是罕见的事件。随着系统规模的不断扩大，它们的可靠性和服务的可用性成为越来越重要的问题。最近的IT支出分析还表明，全球在服务器管理和管理方面的支出已经超过了购买新服务器的成本。传统的反应性故障排除措施和保守的检查点方法往往适得其反，或者可能导致长时间的服务中断。FEMA项目的目标是开发建模和分析方法和工具，以表征高可靠性系统的主动故障管理的系统故障动力学。本FEMA项目从三个方面展开。首先是开发了一个聚合的球形协方差模型，该模型定量地表征了失效动力学。该模型以故障签名概念为中心，该概念将一组操作系统级性能参数和操作级作业分配信息与空间和时间域中不同类型的故障事件相关联。二是统计学习方法在故障预测中的创新应用。在不同的系统范围内，不同的故障类型具有不同的故障动态和不同的训练历史数据量；不同的预测指标对预测粒度提出了不同的要求。各种监督、无监督和强化学习算法在不同的场景中找到了它们的应用。第三是开发用于离线评估的系统可靠性跟踪和用于生产系统在线预测的方法。跟踪不仅包含故障事件的日志，还包含它们相应的操作上下文，这是获得高预测精度所必需的。

In large-scale computer systems, component failures are no longer rare events. As the scale of the systems continues to increase, their reliability and service availability become an increasingly critical concern. Recent IT expenditure analyses also show that the worldwide spending in server management and administration has surpassed the cost of new server acquisition. Conventional reactive trouble-shooting measures and conservative check-pointing approaches are often counter-productive or may cause a long time service disruption. The goal of this FEMA project is to develop modeling and analytical methodologies and tools to characterize the systems failure dynamics for proactive failure management in highly dependable systems.This FEMA project is carried out in three aspects. First is the development of an aggregated spherical covariance model that characterizes the failure dynamics quantitatively. The model centers on a failure signature concept that correlates a group of OS-level performance parameters and operation-level job allocation information to different types of fault events in both space and time domains. Second is an innovative application of statistical learning methods for failure prediction. Different failures types in different system scopes have different failure dynamics and different amount of history data for training; different prediction metrics pose different requirements for prediction granularity. Various supervised, unsupervised, and reinforcement learning algorithms find their applications in different scenarios. Third is the development of system reliability traces for offline evaluation and a methodology for online prediction in production systems. The trace not only contains a log of failure events, but also their corresponding operational contexts that are necessary for attaining high prediction accuracy.