OAC Core: Improving Data Integrity for HPC Datasets using Sparsity Profile

科学家们进行分析，依靠大规模模拟在气候、能源、量子物理等多个科学领域实现突破。随着系统复杂性的增加，未来的大规模系统及其生成、处理、存储和传输的数据会受到越来越高的软错误或静默数据损坏的影响。重要的是，这些静默泄露的数据可能不会被发现，因为当前的高性能计算(HPC)软件堆栈在很大程度上缺乏通知科学家静默数据损坏的机制，这可能会对他们的科学解释的完整性产生不利影响。为了应对HPC系统中的静默数据损坏，该项目引入了高效且经济实惠的机制来监控和检测软错误。通过使用无监督错误检测，该项目增加了科学家对极端规模科学模拟和数据分析的信心，从而推进了解决一些世界上最复杂的当代问题所需的数据密集型科学发现，如预测恶劣天气条件、设计新材料、使新能源变得实用等。该项目的方法也适用于通用计算系统，提高了传统计算和物联网设备的安全性和可靠性。本研究应用压缩感知和机器学习，特别是一种无监督的方法，准确检测当前和未来高性能计算系统中的软、硬件错误。通过不需要改变现有基础设施的硬件辅助数据收集机制，通过压缩感测有效地获得与原始数据集相对应的紧凑表示。这与时空异常检测模型一起使用，用于现场表征软错误和由硬件故障引起的错误，检测偏离可接受范围的异常。该方法内置于科学工作流程中，无需修改或定制应用程序即可与应用程序无缝运行。使用科学工作流在多个HPC平台上验证该机制，使科学家能够以更高的信任级别分析和验证他们的数据集。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Scientists conduct analyses that rely on large-scale simulations to achieve breakthroughs in multiple scientific domains, such as climate, energy, quantum physics, and more. As system complexity increases, future large-scale systems and the data generated, processed, stored, and transmitted by them are subject to increasingly higher occurrences of soft errors or silent data corruption. Importantly, this silently compromised data may go undetected because current High-Performance Computing (HPC) software stacks largely lack mechanisms to inform scientists of silent data corruption that could adversely affect the integrity of their scientific interpretation. In order to combat silent data corruption in HPC systems, this project introduces highly efficient and cost-effective mechanisms to monitor and detect soft errors. Through the use of unsupervised error detection, this project increases scientists’ confidence in extreme-scale scientific simulations and data analyses, which advance the data-intensive science discovery needed to solve some of the world’s most complex contemporary problems, such as predicting severe weather conditions, designing new materials, making new energy sources pragmatic, and others. The methodologies of this project are also applicable to general-purpose computing systems, increasing security and reliability on traditional computing and Internet of Things devices.This research applies compressive sensing and machine learning, especially an unsupervised approach, to accurately detect soft and hardware errors in current and future HPC systems. A compact representation that corresponds to the original dataset is efficiently obtained through compressive sensing coupled with a hardware-assisted data collection mechanism that requires no changes to existing infrastructure. This is used with a spatiotemporal anomaly detection model for in situ characterization of soft errors and errors caused by a hardware malfunction, detecting anomalies deviating from acceptable ranges. The approach is built into the scientific workflow and operates seamlessly with the application without requiring application modification or customization. Validation of the mechanism across multiple HPC platforms using scientific workflows allows scientists to analyze and verify their datasets with increased levels of trust.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.