Application Based Fault Tolerance in High Performance Computing Applications

当我们向百亿亿级系统发展时，故障发生的可能性随着系统中组件的数量而增加。其中一些错误，如软错误（Soft Errors， SE），可能会给数据引入噪声，这取决于系统，可能无法纠正或检测到计算损坏的含义，并可能返回无效的结果。在硬件中实现的大多数常见FT技术使用错误纠正码（ECC）方法，它总是可以纠正任何可纠正的检测到的错误。硬件实现允许以额外的硬件复杂性和内存带宽为代价最小化运行时性能开销。这些实现在额外的计算和内存传输过程中也需要更多的能量，并且由于目前的超级计算机已经消耗超过10MW（足够为一个小镇供电），因此去除这些额外的硬件可以提高百亿亿次级系统的能源效率。我们研究了FT技术的高性能软件替代方案，它们具有明显的优势，因为它们不需要额外的硬件，这在Exascale上将被证明是非常有益的。基于应用程序的容错（ABFT）技术还为故障发生时如何处理故障提供了更大的灵活性，这可以带来更高的性能。通过对高性能计算（HPC）中常见的计算模式和通信模式的研究，提出了新的容错方法。abft技术允许应用程序决定一个特定的错误是需要纠正还是可以忽略，例如，双精度浮点数尾数的低有效位的位翻转可能在算法的几次迭代后收敛到正确的值，因此不需要纠错。ABFT也可用于为不提供ECC功能的硬件提供1 FT，例如嵌入式处理器或消费级gpu。即使硬件确实提供FT，它通常也可以被关闭，而使用abft将释放硬件所需的资源，例如内存和内存带宽，这将提高应用程序的性能。我们特别研究了hpc小矮人的新ABFT技术，并应用信息和编码理论来推导可以检测和纠正（多重）错误的创新方法。然后，我们研究适用于小矮人子集的技术，这些技术是高度优化的，可以包含在软件库中，以便它们随时可以使用。硬件ft技术的一大好处是，它不需要用户为了保护应用程序而更改代码，而ABFT技术依赖于应用程序，通常需要更改应用程序的源代码。为了缓解这个问题，我们使用机器学习等技术研究应用程序的故障脆弱部分的自动检测，然后在应用程序的编译过程中应用ABFT技术。这种方法最大限度地减少了程序员适应这些FT技术并使其应用程序具有容错性所需的努力。

As we are moving towards Exascale systems, the probability of faults occurring increases with the number ofcomponents in the system. Some of these faults, such as Soft Errors (SE), can introduce noise to the data, whichdepending on the system, might be impossible to correct or detect meaning that the computation is corruptedand can potentially return invalid results. Most common FT techniques implemented in hardware use ErrorCorrecting Codes (ECC) methods, which can always correct any detected errors that are correctable. Ahardware implementation allows to minimise the runtime performance overhead at the cost of additionalhardware complexity and memory bandwidth. These implementations also need more energy during additionalcomputations and memory transfers, and as the current supercomputers already consume over 10MW (enoughto power a small town), removing this additional hardware can improve energy efficiency of Exascale systems.We investigate high-performance software alternatives of FT techniques which have a distinct advantage asthey do not require the additional hardware, which at Exascale will prove to be highly beneficial. ApplicationBased Fault Tolerance (ABFT) techniques also bring more flexibility to how faults are dealt with when theyoccur and this can lead to much greater performance. By investigating common High Performance Computing(HPC) computation and communication patterns we derive new methods for Fault Tolerance (FT). ABFTtechniques allow the application to decide whether a particular error needs to be corrected or can be ignored, forexample a bit flip in the less significant bits of the mantissa for a double precision floating point number mayconverge to a correct value after few iterations of the algorithm and hence error correcting is not required.ABFT can also be used to provide 1 FT to hardware that does not provide ECC capabilities, such as embeddedprocessors or consumer GPUs. Even if the hardware does provide FT, it can usually be turned off, and usingABFT instead would free up the resources required by the hardware, such as memory and memory bandwidth,which would improve the applications performance. In particular we investigate new ABFT techniques for theHPC dwarfs and apply Information and Coding theory to derive innovative methods that can detect and correct(multiple) errors. We then look into techniques that apply to a subset of the dwarfs, which are highly optimisedand can included in a software library so that they are ready to use out of the box. A big benefit of hardware FTtechniques is that it does not require the user to change their code in order to protect their application fromfaults, whereas ABFT techniques are application dependent and often require changes to the source code of theapplication. To mitigate this problem we investigate automatic detection of fault vulnerable sections of theapplication using techniques such as machine learning and then apply the ABFT techniques during thecompilation of the application. This approach minimises the efforts required from the programmer to adaptthese FT techniques and make their application fault tolerant.