Collaborative Research: OAC Core: CEAPA: A Systematic Approach to Minimize Compression Error Propagation in HPC Applications

如今的高性能计算(HPC)应用程序会产生海量数据用于后期分析，这给HPC系统带来了巨大的存储和I/O负担。为了显著减轻这一负担，研究人员探索了使用有损压缩技术。有损压缩在有效减小数据量的同时，也给压缩后的数据引入了误差，往往会导致错误的计算结果。因此，科学家们对在他们的科学研究中使用有损压缩犹豫不决。因此，迫切需要开发一种有效的方法来识别将对各种程序的错误影响最小化的压缩策略。该项目旨在开发一种系统的方法，帮助科学家根据他们的HPC程序和目标压缩比自动选择具有最低错误影响的有损压缩算法。它还结合了教育和外联活动，包括学生培训和制定关于可靠数据减少和可靠的高性能计算机系统的新课程。对HPC程序中的压缩误差传播进行建模是具有挑战性的，因为现有的有损压缩器的开发具有不同的原理，这些原理会对不同的HPC数据产生很大不同的压缩误差。该项目包括四个关键任务：(1)开发与常用有损压缩算法的错误模型相结合的准确而高效的错误注入基础设施；(2)设计一种细粒度方法，通过基于压缩数据的数据相关性和生命周期的程序分析和沉积来表征高性能计算程序中的错误传播；(3)利用机器学习技术开发预测模型，以选择对给定程序和压缩比的错误影响最小的压缩策略；以及(4)将该技术与真实世界HPC应用中的特定于域的错误影响度量相结合，并通过选择在相同比率下给出较低错误影响的压缩策略来证明该技术的有效性。该项目不仅对HPC网络基础设施产生了巨大的积极影响，而且还有助于重新定义有损压缩技术的优化，强调效率和错误影响。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Today’s high-performance computing (HPC) applications produce vast volumes of data for post-analysis, presenting a major storage and I/O burden for HPC systems. To significantly reduce this burden, researchers have explored to use lossy compression techniques. While lossy compression can effectively reduce the size of data, it also introduces errors to the compressed data that often lead to incorrect computation results. As a result, scientists hesitate to use lossy compression in their scientific research. Thus, there is a critical need to develop an effective method to identify compression strategies which minimize error impact for a diversity of programs. This project aims to develop a systematic approach that helps scientists automatically select a lossy compression algorithm with the lowest error impact based their HPC programs and target compression ratios. It also integrates educational and outreach activities including student training and development of new curriculum on trustworthy data reduction and dependable HPC systems. Modeling compression error propagation in HPC programs is challenging because existing lossy compressors are developed with distinct principles that generate largely different compression errors on diverse HPC data. This project includes four key thrusts: (1) developing an accurate and efficient fault injection infrastructure that integrates with the fault models of commonly used lossy compression algorithms; (2) designing a fine-grained approach to characterize error propagation in HPC programs through program analysis and deposition based on the data dependencies and life cycle of compressed data; (3) developing a predictive model using machine learning techniques to select a compression strategy that minimizes the error impact on a given program and compression ratio; and (4) integrating the technique with domain-specific error impact metrics in real-world HPC applications and demonstrates the effectiveness of the technique by selecting compression strategies that give low error impact for the same ratios. Not only this project has an enormous positive impact on HPC cyberinfrastructure, but it also helps redefine the optimization of lossy compression techniques with emphasis on both efficiency and error impact.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.