Enabling data storage for petabyte-level Numerical Weather Prediction

网络文件系统和磁旋转介质现在是HPC/HPDA平台中常见的热层存储形式。在过去十年中，非易失性存储器（包括3DXpoint固态驱动器和存储类存储器设备）的新发展已经被证明在数据密集型应用中是有利的，因为它们在延迟、容量、吞吐量和成本方面提供了新的折衷。表示“高”之义性能对象存储已经出现，为传统的POSIX文件系统及其经常阻碍的状态管理提供了替代范例，一致性要求。这些对象存储中的一些不仅被设计为利用更新和更有效的存储介质及其字节粒度随机访问能力，而且还被用户空间IO框架和持久存储器编程库增强，提供丰富的软件环境，用于在终端中实现高性能数据存储功能，用户应用程序。本研究论文将评估这些发展在数值天气预报的数据密集型领域的影响和适用性，特别是通过调整ECMWF的特定领域的高性能IO存储堆栈，以使用新的对象存储技术，如英特尔DAOS，希捷CORTX，DDN WOS和硬件，如3DXpoint内存。这项工作将与爱丁堡大学的EPCC合作进行。

Networked file systems and magnetic spinning media are nowadays a common form of hot-layer storage in HPC/HPDA platforms. New developments over the past decade in non-volatile memory, including 3DXpoint solid-state drives and storage class memory devices, have been shown to be advantageous in data intensive applications for the new trade-off they offer in latency, capacity, throughput and cost.Alongside these hardware developments, high-performance object stores have emerged offering an alternative paradigm to traditional POSIX file systems and their often hindering state management and consistency requirements. Some of these object stores have not only been designed to leverage the newer and more efficient storage media and their byte-grain random access capability, but also have been augmented by user-space IO frameworks and persistent memory programming libraries, providing a rich software landscape for implementing high-performance data storage functionality in end-user applications.This research thesis will assess the impact and suitability of these developments in the data-intensive field of numerical weather prediction, particularly by adapting ECMWF's domain-specific high-performance IO storage stack to use novel object storage technologies such as Intel DAOS, Seagate CORTX, DDN WOS, and hardware such as 3DXpoint memory. This work will be conducted in collaboration with EPCC at the University of Edinburgh.