Elements: Scaling MetPy to Big Data Workflows in Meteorology and Climate Science

MetPy是一个基于python的大气科学软件包；它为读取数据格式、执行计算和创建可视化提供了现代的、经过良好测试的、特定于领域的软件工具。MetPy建立在一组广泛的社区开发的科学Python工具之上，并利用了诸如持续集成、自动文档生成、屏幕录像和Jupyter笔记本教程等技术。该项目改进了MetPy，以解决当前在支持的数据格式、可伸缩性和运行时性能方面的一些限制；它使MetPy更适合于处理更大的数据集，这些数据集在气候科学和基于集合的建模研究中经常遇到。解决了这些问题，MetPy将继续成为Python用户在大气科学中用于小型和大型数据集的强大工具。当配备了现代化和精心设计的工具时，研究人员将能够以更省时的方式更好地利用大量可用数据。该项目有三个主要目标：实现对大型和小型数据集的有效访问，实现对基于云的数据集的访问，以及为使用MetPy和大数据创建培训资源。为了应对这些挑战，该项目正在为MetPy创建一个基准套件，以量化MetPy在重要工作流中的当前性能，以及由于进一步开发而产生的性能改进。使用性能分析工具，可以使用Python性能工具（如Cython和Numba）识别和优化MetPy中的瓶颈。MetPy也正在被重构，以便更好地与Dask库一起工作，Dask库为Python中的分布式计算提供了便利，并允许MetPy更有效地处理大型数据集。世界气象组织（WMO）已将GRIB（栅格二进制）作为其栅格模式输出的标准格式，并将BUFR（气象数据二进制通用表示形式）作为气象观测数据编码的标准格式。该项目开发的MetPy增强功能将能够读取大型云数据存储（如GRIB和BUFR）中使用的其他数据集。所有这些工作都将在MetPy的在线文档中提供额外的、免费的培训材料。该奖项由先进网络基础设施办公室颁发，由物理和动态气象项目以及美国国家科学基金会地球科学理事会综合协作教育与研究部联合支持。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

MetPy is a Python-based software package for atmospheric science; it provides modern, well-tested, domain-specific software tools for reading data formats, performing calculations, and creating visualizations. MetPy builds upon an extensive set of community-developed scientific Python tools, and leverages technologies such as continuous integration, automated documentation generation, screencasts, and Jupyter notebook tutorials. This project advances MetPy to address some current limitations in supported data formats, scalability, and run-time performance; it makes MetPy more suitable for working on much larger datasets, frequently encountered in climate science and ensemble-based modeling studies. Addressing these areas allows MetPy to continue to be a powerful tool for Python users in the atmospheric sciences for both small and large datasets. When equipped with modern and well-engineered tools, researchers will be able to better utilize the large amounts of data available in a more time-efficient way.This project has three main goals: enabling efficient access to big and small datasets, enabling access to cloud-based datasets, and creating training resources for using MetPy with big data. In tackling these challenges, the project is creating a benchmark suite for MetPy, to quantify the current performance of MetPy in important workflows as well as the performance improvements that occur as a result of further development. Using performance profiling tools, bottlenecks in MetPy are identified and optimized using Python performance tools like Cython and Numba. MetPy is also being refactored to work better with the Dask library, which provides facilities for distributed computing in Python and would allow MetPy to work more effectively with large datasets. The World Meteorological Organization (WMO) has made GRIB (GRIdded Binary) its standard format for gridded model output, and BUFR (Binary Universal Form for the Representation of meteorological data) the standard format to encode meteorological observational data. MetPy enhancements developed in this project will enable reading of additional datasets used in large cloud-based data holdings, such as GRIB and BUFR. All of this work will be featured in additional, freely available training materials in MetPy’s online documentation.This award by the Office of Advanced Cyberinfrastructure is jointly supported by the Physical and Dynamic Meteorology Program and the Division of Integrative and Collaborative Education and Research within the NSF Directorate for Geosciences.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.