SHF: Small: Failure Events Modeling and Analysis for Proactive Management in Highly Dependable Systems

在大型计算机系统中，组件故障不再是罕见的事件。随着系统规模的不断扩大，其可靠性和服务可用性成为越来越重要的问题。最近的信息技术支出分析还表明，全球服务器管理和行政支出已超过购置新服务器的成本。传统的被动式故障排除措施和保守的检查点方法往往适得其反，或可能导致长时间的服务中断。本FEMA项目的目标是开发建模和分析方法和工具，以表征高度可靠系统中的主动故障管理的系统故障动态。首先是一个聚合的球形协方差模型，定量表征故障动力学的发展。该模型集中在一个故障签名的概念，相关的一组操作系统级的性能参数和操作级的作业分配信息，在空间和时间域的不同类型的故障事件。第二是统计学习方法在故障预测中的创新应用。不同系统范围内的不同故障类型具有不同的故障动力学和不同的历史数据量用于训练;不同的预测度量对预测粒度提出不同的要求。各种监督、无监督和强化学习算法在不同的场景中找到了应用。第三是开发离线评估的系统可靠性跟踪和生产系统在线预测的方法。跟踪不仅包含故障事件的日志，而且还包含获得高预测精度所必需的相应操作上下文。

In large-scale computer systems, component failures are no longer rare events. As the scale of the systems continues to increase, their reliability and service availability become an increasingly critical concern. Recent IT expenditure analyses also show that the worldwide spending in server management and administration has surpassed the cost of new server acquisition. Conventional reactive trouble-shooting measures and conservative check-pointing approaches are often counter-productive or may cause a long time service disruption. The goal of this FEMA project is to develop modeling and analytical methodologies and tools to characterize the systems failure dynamics for proactive failure management in highly dependable systems.This FEMA project is carried out in three aspects. First is the development of an aggregated spherical covariance model that characterizes the failure dynamics quantitatively. The model centers on a failure signature concept that correlates a group of OS-level performance parameters and operation-level job allocation information to different types of fault events in both space and time domains. Second is an innovative application of statistical learning methods for failure prediction. Different failures types in different system scopes have different failure dynamics and different amount of history data for training; different prediction metrics pose different requirements for prediction granularity. Various supervised, unsupervised, and reinforcement learning algorithms find their applications in different scenarios. Third is the development of system reliability traces for offline evaluation and a methodology for online prediction in production systems. The trace not only contains a log of failure events, but also their corresponding operational contexts that are necessary for attaining high prediction accuracy.