CRII: OAC: Enabling Quantities-of-Interest Error Control for Trust-Driven Lossy Compression

该奖项全部或部分由《2021年美国救援计划法案》（公法117-2）资助。科学模拟和仪器产生的数据量和速度超过了网络和存储系统。尽管错误控制的有损压缩器已经被用来缓解这些数据问题，但许多科学家仍然不愿意采用它们，因为这些压缩器不能保证从原始数据中得出的下游分析结果的准确性。本项目旨在通过开发一种信任驱动的有损数据压缩基础设施来填补这一空白，该基础设施能够从理论上和实践上严格控制下游分析中的错误，从而促进数据缩减在科学应用中的使用。该项目的成功将通过有效的数据简化促进多学科的科学进步，并有助于解决包括发电、天气预报、材料设计和交通运输在内的重要社会问题。此外，该项目将通过教育和参与活动，包括开发新课程和招募K-12学生，为未来几代科学家和工程师的成长和发展做出贡献。现有的有损压缩技术要么忽略误差量化，要么只对原始数据提供误差控制，从而在从原始数据计算的下游兴趣量（qoi）的结果中留下不确定性。这使许多希望在保留必要信息的同时减少数据量的计算科学家非常担心，从而阻止他们在应用程序中采用有损压缩。本研究将通过三个任务，通过理论与实践的结合来解决这些问题。首先，将开发一种新的理论，使下游qos的误差控制。这将从根本上解决现有的错误控制有损压缩器的可靠性问题，这些压缩器仅对原始数据提供错误控制。其次，在严格分析的基础上，采用保证严格误差控制的优化方法，在相同要求下实现更高的压缩比。第三，通过与先进的压缩框架和基于目标qos的定制并行化的仔细集成，构建可扩展的基础设施，以便充分利用目标qos中现有的压缩算法和计算模式。该项目将使应用科学家能够根据他们的独特需求在他们的数据中存储最有价值的信息，为包括气候学、宇宙学和地震学在内的多个科学学科的新发现创造机会。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Scientific simulations and instruments are producing data at volumes and velocities that overwhelm network and storage systems. Although error-controlled lossy compressors have been employed to mitigate these data issues, many scientists still feel reluctant to adopt them because these compressors provide no guarantee on the accuracy of downstream analysis results derived from raw data. This project aims to fill this gap by developing a trust-driven lossy data compression infrastructure capable of strictly controlling the errors in downstream analysis theoretically and practically to facilitate the use of data reduction in scientific applications. Success of this project will promote the progress of science in multiple disciplines via effective data reduction, and contribute to resolving important societal problems including electric generation, weather forecasting, material design, and transportation. Moreover, this project will contribute to the growth and development of future generations of scientists and engineers through educational and engagement activities, including development of new curriculum and recruitment of K-12 students.Existing lossy compression techniques either overlook error quantification or provide error control only for raw data, leaving uncertainties in the outcome of downstream quantities of interest (QoIs) computed from the raw data. This greatly concerns many computational scientists who wish to reduce their data while preserving necessary information, preventing them from adopting lossy compression in their applications. This research will address these problems through an integration of theory and implementation via three tasks. First, a novel theory enabling error control on downstream QoIs will be developed. This will fundamentally address the trustability issues of existing error controlled lossy compressors that provide error control only on raw data. Second, an optimization method ensuring tight error control will be applied based on rigorous analysis, to achieve higher compression ratios under the same requirements. Third, a scalable infrastructure will be built through a careful integration with advanced compression frameworks and tailored parallelization based on target QoIs, in order to take full advantage of existing compression algorithms and computational patterns in the target QoIs. The project will enable application scientists to store the most valuable information in their data based on their unique needs, creating opportunities for novel findings in multiple scientific disciplines including climatology, cosmology, and seismology.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.