Ceph High-Performance Storage Cluster

点击翻译按钮获取中文摘要

Cutting edge science using cryogenic electron microscopy (cryo-EM), cryo electron tomography (cryo-ET), serial-section scanning electron microscopy as used in high-resolution brain imaging (connectomics), DNA sequencing of genomes and single cells and light sheet microscopy produces ever increasing amounts of data. These data are enabling many new discoveries, but their volume creates signficant technical problems since the data need to be stored, processed and analysed to become useful. At the LMB, we generate multiple petabytes of high-value scientific datasets each year and these datasets often need to be stored, processed and analysed for several months before they can be published and then archived. The very size of these datasets creates unique technical problems in storing and processing them. First, the storage system needs to have enough capacity to be able to hold many datasets at once, since the storage system serves the entire LMB. Second, processing and analysis most of the time use the LMB's parallel compute cluster that requires the storage system to be fast and acessible from hundreds to thousands of compute nodes at the same time, for their parallel access. For these purposes we propose to invest in a large Ceph cluster storage system that fulfils all of these criteria, while being maximally cost effective, and without vendor lock-in. Ceph is comminity driven and used and supported by other scientific organisations with large data requirements such as CERN in Geneva, Switzerland.