CRII: OAC: Enabling Quantities-of-Interest Error Control for Trust-Driven Lossy Compression

该奖项全部或部分由2021年美国救援计划法案（公法117-2）资助。科学模拟和仪器正在以压倒网络和存储系统的数量和速度产生数据。虽然误差控制有损压缩器已被用于缓解这些数据问题，但许多科学家仍然不愿意采用它们，因为这些压缩器无法保证从原始数据中得出的下游分析结果的准确性。本项目旨在填补这一空白，通过开发一个信任驱动的有损数据压缩基础设施，能够严格控制下游分析中的错误，从理论和实践上促进数据约简在科学应用中的使用。该项目的成功将通过有效的数据简化促进多学科的科学进步，并有助于解决重要的社会问题，包括发电，天气预报，材料设计和运输。此外，该项目将通过教育和参与活动，包括开发新课程和招募K-12学生，为未来几代科学家和工程师的成长和发展做出贡献。现有的有损压缩技术要么忽略了误差量化，要么只为原始数据提供误差控制，从而使从原始数据计算的下游感兴趣量（QoI）的结果存在不确定性。这极大地影响了许多计算科学家，他们希望减少数据，同时保留必要的信息，防止他们在应用程序中采用有损压缩。本研究将通过三个任务的理论和实施的整合来解决这些问题。首先，一个新的理论，使错误控制下游的QoS将被开发。这将从根本上解决现有错误控制有损压缩器的可信性问题，这些压缩器仅对原始数据提供错误控制。其次，将基于严格的分析应用确保严格的错误控制的优化方法，以在相同的要求下实现更高的压缩比。第三，将通过与高级压缩框架的仔细集成和基于目标QoS的定制并行化来构建可扩展的基础设施，以便充分利用目标QoS中的现有压缩算法和计算模式。该项目将使应用科学家能够根据他们的独特需求将最有价值的信息存储在他们的数据中，为包括气候学，宇宙学和地震学在内的多个科学学科的新发现创造机会。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估而被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Scientific simulations and instruments are producing data at volumes and velocities that overwhelm network and storage systems. Although error-controlled lossy compressors have been employed to mitigate these data issues, many scientists still feel reluctant to adopt them because these compressors provide no guarantee on the accuracy of downstream analysis results derived from raw data. This project aims to fill this gap by developing a trust-driven lossy data compression infrastructure capable of strictly controlling the errors in downstream analysis theoretically and practically to facilitate the use of data reduction in scientific applications. Success of this project will promote the progress of science in multiple disciplines via effective data reduction, and contribute to resolving important societal problems including electric generation, weather forecasting, material design, and transportation. Moreover, this project will contribute to the growth and development of future generations of scientists and engineers through educational and engagement activities, including development of new curriculum and recruitment of K-12 students.Existing lossy compression techniques either overlook error quantification or provide error control only for raw data, leaving uncertainties in the outcome of downstream quantities of interest (QoIs) computed from the raw data. This greatly concerns many computational scientists who wish to reduce their data while preserving necessary information, preventing them from adopting lossy compression in their applications. This research will address these problems through an integration of theory and implementation via three tasks. First, a novel theory enabling error control on downstream QoIs will be developed. This will fundamentally address the trustability issues of existing error controlled lossy compressors that provide error control only on raw data. Second, an optimization method ensuring tight error control will be applied based on rigorous analysis, to achieve higher compression ratios under the same requirements. Third, a scalable infrastructure will be built through a careful integration with advanced compression frameworks and tailored parallelization based on target QoIs, in order to take full advantage of existing compression algorithms and computational patterns in the target QoIs. The project will enable application scientists to store the most valuable information in their data based on their unique needs, creating opportunities for novel findings in multiple scientific disciplines including climatology, cosmology, and seismology.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.