Elements: Scaling MetPy to Big Data Workflows in Meteorology and Climate Science

MetPy 是一个基于 Python 的大气科学软件包；它提供了现代的、经过充分测试的、特定领域的软件工具，用于读取数据格式、执行计算和创建可视化。 MetPy 基于社区开发的一系列广泛的科学 Python 工具而构建，并利用持续集成、自动文档生成、截屏视频和 Jupyter Notebook 教程等技术。 该项目推动 MetPy 解决当前支持的数据格式、可扩展性和运行时性能方面的一些限制；它使 MetPy 更适合处理更大的数据集，这些数据集在气候科学和基于集合的建模研究中经常遇到。解决这些领域的问题使得 MetPy 能够继续成为大气科学领域 Python 用户处理小型和大型数据集的强大工具。当配备现代且精心设计的工具时，研究人员将能够以更省时的方式更好地利用大量可用数据。该项目有三个主要目标：实现对大小数据集的高效访问，实现对基于云的数据集的访问，以及创建使用 MetPy 处理大数据的培训资源。 为了应对这些挑战，该项目正在为 MetPy 创建基准套件，以量化 MetPy 在重要工作流程中的当前性能以及进一步开发带来的性能改进。使用性能分析工具，可以使用 Cython 和 Numba 等 Python 性能工具来识别和优化 MetPy 中的瓶颈。 MetPy 还进行了重构，以便更好地与 Dask 库配合使用，该库为 Python 中的分布式计算提供了设施，并使 MetPy 能够更有效地处理大型数据集。 世界气象组织（WMO）已将GRIB（GRIdded Binary）作为网格模型输出的标准格式，并将BUFR（气象数据表示的二进制通用格式）作为编码气象观测数据的标准格式。该项目中开发的 MetPy 增强功能将能够读取大型基于云的数据持有中使用的其他数据集，例如 GRIB 和 BUFR。 所有这些工作都将包含在 MetPy 在线文档中免费提供的额外培训材料中。该奖项由先进网络基础设施办公室颁发，并得到物理和动态气象学计划以及 NSF 地球科学理事会综合与协作教育与研究部的共同支持。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

MetPy is a Python-based software package for atmospheric science; it provides modern, well-tested, domain-specific software tools for reading data formats, performing calculations, and creating visualizations. MetPy builds upon an extensive set of community-developed scientific Python tools, and leverages technologies such as continuous integration, automated documentation generation, screencasts, and Jupyter notebook tutorials. This project advances MetPy to address some current limitations in supported data formats, scalability, and run-time performance; it makes MetPy more suitable for working on much larger datasets, frequently encountered in climate science and ensemble-based modeling studies. Addressing these areas allows MetPy to continue to be a powerful tool for Python users in the atmospheric sciences for both small and large datasets. When equipped with modern and well-engineered tools, researchers will be able to better utilize the large amounts of data available in a more time-efficient way.This project has three main goals: enabling efficient access to big and small datasets, enabling access to cloud-based datasets, and creating training resources for using MetPy with big data. In tackling these challenges, the project is creating a benchmark suite for MetPy, to quantify the current performance of MetPy in important workflows as well as the performance improvements that occur as a result of further development. Using performance profiling tools, bottlenecks in MetPy are identified and optimized using Python performance tools like Cython and Numba. MetPy is also being refactored to work better with the Dask library, which provides facilities for distributed computing in Python and would allow MetPy to work more effectively with large datasets. The World Meteorological Organization (WMO) has made GRIB (GRIdded Binary) its standard format for gridded model output, and BUFR (Binary Universal Form for the Representation of meteorological data) the standard format to encode meteorological observational data. MetPy enhancements developed in this project will enable reading of additional datasets used in large cloud-based data holdings, such as GRIB and BUFR. All of this work will be featured in additional, freely available training materials in MetPy’s online documentation.This award by the Office of Advanced Cyberinfrastructure is jointly supported by the Physical and Dynamic Meteorology Program and the Division of Integrative and Collaborative Education and Research within the NSF Directorate for Geosciences.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.