CPA: A Hybrid Fault Tolerant Approach for High-End Computing

几乎所有的科学和工程领域都依赖于计算机领域的根本性进步。高端计算(HEC)在各个科学领域的模拟使我们能够了解我们周围的世界。不幸的是，港灯以缺乏持续的性能和可靠性而闻名。它的全系统故障率随着组件数量的增加而显著增加。HEC的传统容错方法--检查点--成本高昂，而且会引发恶性循环。港灯不断增加的复杂性加剧了这种恶化。新一代HEC必须采用一种新的容错方法。在这项研究中，PI为HEC提出了一种新的混合容错(HFT)方法，该方法结合了长期和短期技术来改进故障管理。长期预测基于历史数据对故障的可能性进行建模，从而通过智能地将作业映射到可用资源来促进故障感知调度。短期预测可诊断异常运行时事件的根本原因，并触发作业动态重新调度以将正在运行的作业从这些麻烦的资源中移走。长期支持和短期支持相辅相成，其中故障感知调度防止非活动作业(即尚未调度的作业)从长期故障模型中很好地捕获的故障中获得，并且故障感知重新调度使得活动作业(即已经调度并运行的作业)能够避免可能不遵循任何长期模式但可以在运行时被发现的不规则故障(例如突然的硬件和软件错误)。综合的长期和短期方法有助于更好地了解故障趋势和模式，从而提高港灯的系统生产率。

Virtually all fields of science and engineering depend on fundamental advances in computing. High- End Computing (HEC) simulations in various areas of science enable to understand the world around us. Unfortunately, HEC is known for lack of sustained performance and reliability. Its system-wide failure rate increases significantly with the growing number of components. The conventional method for fault tolerance in HEC, checkpointing, is costly and triggers a cycle of deterioration. This deterioration is fueled by ever-increasing HEC complexity. A new fault tolerant approach is a must for next generation HEC. In this research, the PIs propose a novel Hybrid Fault Tolerant (HFT) approach for HEC that combines long-term and short-term techniques to improve fault management. Long-term prediction models the possibility of faults based on historical data, and consequently facilitates failure-aware scheduling by intelligently mapping jobs to available resources. Short-term prediction diagnoses the root causes of unusual runtime events, and triggers job rescheduling on-the-fly to move running jobs away from these troublesome resources. The long-term support and the short-term support complement each other, where failure-aware scheduling prevents inactive jobs (i.e. the jobs that are not scheduled yet) from the failures that are well captured in the long-term failure models and failure-aware rescheduling enables active jobs (i.e. the jobs that are already scheduled and running) to avoid irregular failures that may not follow any long-term pattern but can be discovered at runtime (e.g. sudden hardware and software errors). The integrated long-term and short-term approach promotes a better understanding of failure trends and modes and consequently improves system productivity in HEC.