Enabling data storage for petabyte-level Numerical Weather Prediction

网络文件系统和磁旋转介质是目前HPC/HPDA平台中热层存储的常见形式。过去十年非易失性存储器的新发展，包括3DXpoint固态驱动器和存储级存储器设备，已被证明在数据密集型应用中具有优势，因为它们在延迟、容量、吞吐量和成本方面提供了新的权衡。随着这些硬件的发展，高性能对象存储已经出现，为传统的POSIX文件系统及其通常阻碍状态管理和一致性需求提供了另一种范例。其中一些对象存储不仅被设计为利用更新和更高效的存储介质及其字节粒随机访问能力，而且还被用户空间IO框架和持久内存编程库增强，为在最终用户应用程序中实现高性能数据存储功能提供了丰富的软件环境。本研究论文将评估这些发展在数值天气预报数据密集型领域的影响和适用性，特别是通过调整ECMWF特定领域的高性能IO存储堆栈来使用新的对象存储技术，如英特尔DAOS，希捷CORTX， DDN WOS，以及3DXpoint内存等硬件。这项工作将与爱丁堡大学的EPCC合作进行。

Networked file systems and magnetic spinning media are nowadays a common form of hot-layer storage in HPC/HPDA platforms. New developments over the past decade in non-volatile memory, including 3DXpoint solid-state drives and storage class memory devices, have been shown to be advantageous in data intensive applications for the new trade-off they offer in latency, capacity, throughput and cost.Alongside these hardware developments, high-performance object stores have emerged offering an alternative paradigm to traditional POSIX file systems and their often hindering state management and consistency requirements. Some of these object stores have not only been designed to leverage the newer and more efficient storage media and their byte-grain random access capability, but also have been augmented by user-space IO frameworks and persistent memory programming libraries, providing a rich software landscape for implementing high-performance data storage functionality in end-user applications.This research thesis will assess the impact and suitability of these developments in the data-intensive field of numerical weather prediction, particularly by adapting ECMWF's domain-specific high-performance IO storage stack to use novel object storage technologies such as Intel DAOS, Seagate CORTX, DDN WOS, and hardware such as 3DXpoint memory. This work will be conducted in collaboration with EPCC at the University of Edinburgh.