Enabling data storage for petabyte-level Numerical Weather Prediction

点击翻译按钮获取中文摘要

Networked file systems and magnetic spinning media are nowadays a common form of hot-layer storage in HPC/HPDA platforms. New developments over the past decade in non-volatile memory, including 3DXpoint solid-state drives and storage class memory devices, have been shown to be advantageous in data intensive applications for the new trade-off they offer in latency, capacity, throughput and cost.Alongside these hardware developments, high-performance object stores have emerged offering an alternative paradigm to traditional POSIX file systems and their often hindering state management and consistency requirements. Some of these object stores have not only been designed to leverage the newer and more efficient storage media and their byte-grain random access capability, but also have been augmented by user-space IO frameworks and persistent memory programming libraries, providing a rich software landscape for implementing high-performance data storage functionality in end-user applications.This research thesis will assess the impact and suitability of these developments in the data-intensive field of numerical weather prediction, particularly by adapting ECMWF's domain-specific high-performance IO storage stack to use novel object storage technologies such as Intel DAOS, Seagate CORTX, DDN WOS, and hardware such as 3DXpoint memory. This work will be conducted in collaboration with EPCC at the University of Edinburgh.