OAC Core: Improving Data Integrity for HPC Datasets using Sparsity Profile

科学家们进行分析，依靠大规模模拟来实现多个科学领域的突破，如气候、能源、量子物理等。随着系统复杂性的增加，未来的大型系统及其生成、处理、存储和传输的数据将越来越容易出现软错误或静默数据损坏。重要的是，这种被悄无声息地破坏的数据可能不会被发现，因为当前的高性能计算（HPC）软件堆栈在很大程度上缺乏告知科学家可能对其科学解释的完整性产生不利影响的无声数据损坏的机制。为了打击高性能计算系统中的静默数据损坏，本项目引入了高效和经济的机制来监测和检测软错误。通过使用无监督错误检测，该项目提高了科学家对极端尺度科学模拟和数据分析的信心，从而推进了解决一些世界上最复杂的当代问题所需的数据密集型科学发现，如预测恶劣天气条件、设计新材料、使新能源实用等。该项目的方法也适用于通用计算系统，提高传统计算和物联网设备的安全性和可靠性。本研究应用压缩感知和机器学习，特别是一种无监督的方法，来准确检测当前和未来HPC系统中的软、硬件错误。通过压缩感知和硬件辅助的数据收集机制（不需要更改现有基础设施），可以有效地获得与原始数据集对应的紧凑表示。这与时空异常检测模型一起使用，用于对软误差和由硬件故障引起的误差进行原位表征，检测偏离可接受范围的异常。该方法内置于科学工作流程中，无需对应用程序进行修改或自定义，即可与应用程序无缝操作。使用科学工作流程跨多个HPC平台的机制验证使科学家能够以更高的信任水平分析和验证他们的数据集。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Scientists conduct analyses that rely on large-scale simulations to achieve breakthroughs in multiple scientific domains, such as climate, energy, quantum physics, and more. As system complexity increases, future large-scale systems and the data generated, processed, stored, and transmitted by them are subject to increasingly higher occurrences of soft errors or silent data corruption. Importantly, this silently compromised data may go undetected because current High-Performance Computing (HPC) software stacks largely lack mechanisms to inform scientists of silent data corruption that could adversely affect the integrity of their scientific interpretation. In order to combat silent data corruption in HPC systems, this project introduces highly efficient and cost-effective mechanisms to monitor and detect soft errors. Through the use of unsupervised error detection, this project increases scientists’ confidence in extreme-scale scientific simulations and data analyses, which advance the data-intensive science discovery needed to solve some of the world’s most complex contemporary problems, such as predicting severe weather conditions, designing new materials, making new energy sources pragmatic, and others. The methodologies of this project are also applicable to general-purpose computing systems, increasing security and reliability on traditional computing and Internet of Things devices.This research applies compressive sensing and machine learning, especially an unsupervised approach, to accurately detect soft and hardware errors in current and future HPC systems. A compact representation that corresponds to the original dataset is efficiently obtained through compressive sensing coupled with a hardware-assisted data collection mechanism that requires no changes to existing infrastructure. This is used with a spatiotemporal anomaly detection model for in situ characterization of soft errors and errors caused by a hardware malfunction, detecting anomalies deviating from acceptable ranges. The approach is built into the scientific workflow and operates seamlessly with the application without requiring application modification or customization. Validation of the mechanism across multiple HPC platforms using scientific workflows allows scientists to analyze and verify their datasets with increased levels of trust.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.