CRII: OAC: Enabling Quantities-of-Interest Error Control for Trust-Driven Lossy Compression

该奖项的全部或部分资金来自《2021年美国救援计划法案》(公法117-2)。科学模拟和仪器产生的数据量和速度使网络和存储系统不堪重负。尽管误差控制有损压缩器已经被用来缓解这些数据问题，但许多科学家仍然不愿采用它们，因为这些压缩器不能保证从原始数据得出的下游分析结果的准确性。该项目旨在通过开发信任驱动的有损数据压缩基础设施来填补这一空白，该基础设施能够在理论和实践上严格控制下游分析中的错误，从而促进数据约简在科学应用中的使用。该项目的成功将通过有效的数据减少促进多学科的科学进步，并有助于解决包括发电、天气预报、材料设计和交通在内的重要社会问题。此外，该项目将通过教育和参与活动，包括开发新课程和招募K-12学生，促进未来几代科学家和工程师的成长和发展。现有的有损压缩技术要么忽略了错误量化，要么只为原始数据提供错误控制，从而使根据原始数据计算的下游兴趣量(QOI)的结果存在不确定性。这让许多计算科学家非常担心，他们希望在减少数据的同时保留必要的信息，防止他们在应用程序中采用有损压缩。本研究将通过三项任务，通过理论和实施相结合的方式来解决这些问题。首先，将开发一种新的理论来实现对下游QOI的差错控制。这将从根本上解决现有错误控制有损压缩器的可信性问题，这些压缩器仅对原始数据提供错误控制。其次，将在严格分析的基础上，采用确保严格差错控制的优化方法，在相同要求的情况下实现更高的压缩比。第三，通过与先进的压缩框架和基于目标QOI的定制并行化的仔细集成，构建可扩展的基础设施，以充分利用目标QOI中现有的压缩算法和计算模式。该项目将使应用科学家能够根据他们的独特需求将最有价值的信息存储在他们的数据中，为包括气候学、宇宙学和地震学在内的多个科学学科的新发现创造机会。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Scientific simulations and instruments are producing data at volumes and velocities that overwhelm network and storage systems. Although error-controlled lossy compressors have been employed to mitigate these data issues, many scientists still feel reluctant to adopt them because these compressors provide no guarantee on the accuracy of downstream analysis results derived from raw data. This project aims to fill this gap by developing a trust-driven lossy data compression infrastructure capable of strictly controlling the errors in downstream analysis theoretically and practically to facilitate the use of data reduction in scientific applications. Success of this project will promote the progress of science in multiple disciplines via effective data reduction, and contribute to resolving important societal problems including electric generation, weather forecasting, material design, and transportation. Moreover, this project will contribute to the growth and development of future generations of scientists and engineers through educational and engagement activities, including development of new curriculum and recruitment of K-12 students.Existing lossy compression techniques either overlook error quantification or provide error control only for raw data, leaving uncertainties in the outcome of downstream quantities of interest (QoIs) computed from the raw data. This greatly concerns many computational scientists who wish to reduce their data while preserving necessary information, preventing them from adopting lossy compression in their applications. This research will address these problems through an integration of theory and implementation via three tasks. First, a novel theory enabling error control on downstream QoIs will be developed. This will fundamentally address the trustability issues of existing error controlled lossy compressors that provide error control only on raw data. Second, an optimization method ensuring tight error control will be applied based on rigorous analysis, to achieve higher compression ratios under the same requirements. Third, a scalable infrastructure will be built through a careful integration with advanced compression frameworks and tailored parallelization based on target QoIs, in order to take full advantage of existing compression algorithms and computational patterns in the target QoIs. The project will enable application scientists to store the most valuable information in their data based on their unique needs, creating opportunities for novel findings in multiple scientific disciplines including climatology, cosmology, and seismology.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.