Elements: Scaling MetPy to Big Data Workflows in Meteorology and Climate Science

MetPy是一个基于Python的大气科学软件包；它提供了现代的、经过充分测试的、特定于领域的软件工具，用于读取数据格式、执行计算和创建可视化。MetPy建立在社区开发的大量科学工具的基础上，并利用持续集成、自动文档生成、截屏视频和Jupyter笔记本教程等技术。该项目推进了MetPy，以解决当前在支持的数据格式、可伸缩性和运行时性能方面的一些限制；它使MetPy更适合处理气候科学和基于集合的建模研究中经常遇到的更大的数据集。解决这些问题使MetPy继续成为大气层科学领域中小型和大型数据集的Python用户的强大工具。当配备现代和精心设计的工具时，研究人员将能够更好地以更省时的方式利用可用的海量数据。该项目有三个主要目标：实现对大数据集和小数据集的高效访问，实现对基于云的数据集的访问，以及创建将MetPy与大数据一起使用的培训资源。为了应对这些挑战，该项目正在为MetPy创建一个基准套件，以量化MetPy在重要工作流程中的当前表现以及由于进一步发展而出现的性能改进。通过使用性能分析工具，可以识别和优化MetPy中的瓶颈，并使用Cython和Numba等Python性能工具进行优化。MetPy也正在进行重构，以便更好地与Dask库一起工作，该库提供了使用Python语言进行分布式计算的工具，并将使MetPy能够更有效地处理大型数据集。世界气象组织(WMO)已将GRIB(网格二进制)作为网格模式输出的标准格式，BUFR(用于表示气象数据的二进制通用格式)作为对气象观测数据进行编码的标准格式。此项目中开发的MetPy增强功能将支持读取大型基于云的数据存储(如GRIB和BUFR)中使用的其他数据集。所有这些工作都将在MetPy的在线文件中免费提供额外的培训材料。这一奖项由高级网络基础设施办公室颁发，由物理和动态气象学项目以及NSF地球科学局内的综合和合作教育与研究部门共同支持。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

MetPy is a Python-based software package for atmospheric science; it provides modern, well-tested, domain-specific software tools for reading data formats, performing calculations, and creating visualizations. MetPy builds upon an extensive set of community-developed scientific Python tools, and leverages technologies such as continuous integration, automated documentation generation, screencasts, and Jupyter notebook tutorials. This project advances MetPy to address some current limitations in supported data formats, scalability, and run-time performance; it makes MetPy more suitable for working on much larger datasets, frequently encountered in climate science and ensemble-based modeling studies. Addressing these areas allows MetPy to continue to be a powerful tool for Python users in the atmospheric sciences for both small and large datasets. When equipped with modern and well-engineered tools, researchers will be able to better utilize the large amounts of data available in a more time-efficient way.This project has three main goals: enabling efficient access to big and small datasets, enabling access to cloud-based datasets, and creating training resources for using MetPy with big data. In tackling these challenges, the project is creating a benchmark suite for MetPy, to quantify the current performance of MetPy in important workflows as well as the performance improvements that occur as a result of further development. Using performance profiling tools, bottlenecks in MetPy are identified and optimized using Python performance tools like Cython and Numba. MetPy is also being refactored to work better with the Dask library, which provides facilities for distributed computing in Python and would allow MetPy to work more effectively with large datasets. The World Meteorological Organization (WMO) has made GRIB (GRIdded Binary) its standard format for gridded model output, and BUFR (Binary Universal Form for the Representation of meteorological data) the standard format to encode meteorological observational data. MetPy enhancements developed in this project will enable reading of additional datasets used in large cloud-based data holdings, such as GRIB and BUFR. All of this work will be featured in additional, freely available training materials in MetPy’s online documentation.This award by the Office of Advanced Cyberinfrastructure is jointly supported by the Physical and Dynamic Meteorology Program and the Division of Integrative and Collaborative Education and Research within the NSF Directorate for Geosciences.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.