Application Based Fault Tolerance in High Performance Computing Applications

随着我们走向亿级系统，故障发生的概率随着系统中组件的数量增加而增加。其中一些错误，如软错误(SE)，可能会向数据引入噪声，根据系统的不同，这些噪声可能无法纠正或检测到计算已损坏，并可能返回无效结果。硬件中实现的大多数常见FT技术使用纠错码(ECC)方法，该方法总是可以纠正任何检测到的可纠正的错误。硬件实现允许以额外的硬件复杂性和内存带宽为代价来最小化运行时性能开销。在额外的计算和内存传输过程中，这些实现还需要更多的能量，并且由于当前的超级计算机已经消耗超过10 mW(足以为一个小镇供电)，因此移除这些额外的硬件可以提高艾斯卡系统的能效。我们研究了FT技术的高性能软件替代方案，它们具有明显的优势，因为它们不需要额外的硬件，这在艾斯卡将被证明是非常有益的。基于应用程序的容错(ABFT)技术还为故障的处理带来了更大的灵活性，这可以带来更高的性能。通过研究常见的高性能计算(HPC)计算和通信模式，我们得到了新的容错(FT)方法。ABFT技术允许应用程序决定是否需要纠正或可以忽略特定的错误，例如，双精度浮点数的尾数的较低有效位的比特翻转可能在算法的几次迭代后收敛到正确值，因此不需要纠错。ABFT还可以用于向不提供ECC能力的硬件提供1FT，例如嵌入式处理器或消费者GPU。即使硬件确实提供FT，通常也可以将其关闭，而使用ABFT将释放硬件所需的资源，如内存和内存带宽，这将提高应用程序的性能。特别是，我们研究了用于HPC矮星的新的ABFT技术，并应用信息和编码理论来推导出能够检测和纠正(多个)错误的创新方法。然后，我们研究适用于矮人子集的技术，这些技术经过高度优化，可以包含在软件库中，以便随时可以使用。硬件FT技术的一大好处是，它不需要用户为了保护他们的应用程序免受故障而更改他们的代码，而ABFT技术依赖于应用程序，并且通常需要更改应用程序的源代码。为了缓解这个问题，我们研究了使用机器学习等技术自动检测应用程序的故障易受攻击部分，然后在应用程序的编译过程中应用ABFT技术。这种方法最大限度地减少了程序员适应这些FT技术并使其应用程序容错所需的工作。

As we are moving towards Exascale systems, the probability of faults occurring increases with the number ofcomponents in the system. Some of these faults, such as Soft Errors (SE), can introduce noise to the data, whichdepending on the system, might be impossible to correct or detect meaning that the computation is corruptedand can potentially return invalid results. Most common FT techniques implemented in hardware use ErrorCorrecting Codes (ECC) methods, which can always correct any detected errors that are correctable. Ahardware implementation allows to minimise the runtime performance overhead at the cost of additionalhardware complexity and memory bandwidth. These implementations also need more energy during additionalcomputations and memory transfers, and as the current supercomputers already consume over 10MW (enoughto power a small town), removing this additional hardware can improve energy efficiency of Exascale systems.We investigate high-performance software alternatives of FT techniques which have a distinct advantage asthey do not require the additional hardware, which at Exascale will prove to be highly beneficial. ApplicationBased Fault Tolerance (ABFT) techniques also bring more flexibility to how faults are dealt with when theyoccur and this can lead to much greater performance. By investigating common High Performance Computing(HPC) computation and communication patterns we derive new methods for Fault Tolerance (FT). ABFTtechniques allow the application to decide whether a particular error needs to be corrected or can be ignored, forexample a bit flip in the less significant bits of the mantissa for a double precision floating point number mayconverge to a correct value after few iterations of the algorithm and hence error correcting is not required.ABFT can also be used to provide 1 FT to hardware that does not provide ECC capabilities, such as embeddedprocessors or consumer GPUs. Even if the hardware does provide FT, it can usually be turned off, and usingABFT instead would free up the resources required by the hardware, such as memory and memory bandwidth,which would improve the applications performance. In particular we investigate new ABFT techniques for theHPC dwarfs and apply Information and Coding theory to derive innovative methods that can detect and correct(multiple) errors. We then look into techniques that apply to a subset of the dwarfs, which are highly optimisedand can included in a software library so that they are ready to use out of the box. A big benefit of hardware FTtechniques is that it does not require the user to change their code in order to protect their application fromfaults, whereas ABFT techniques are application dependent and often require changes to the source code of theapplication. To mitigate this problem we investigate automatic detection of fault vulnerable sections of theapplication using techniques such as machine learning and then apply the ABFT techniques during thecompilation of the application. This approach minimises the efforts required from the programmer to adaptthese FT techniques and make their application fault tolerant.