HDF5-Based I/O Optimization for Extragalactic HI Data Pipeline of FAST
HDF5-Based I/O Optimization for Extragalactic HI Data Pipeline of FAST
复制标题
DOI：
10.1007/978-3-030-38961-1_55
复制
发表时间：
2019-12
期刊：
影响因子：
--
通讯作者：
Yiming Ji;Ce Yu;Jian Xiao;Shanjian Tang;Hao Wang;Bo Zhang
中科院分区：
文献类型：
--
作者：
Yiming Ji;Ce Yu;Jian Xiao;Shanjian Tang;Hao Wang;Bo Zhang
500米口径球面射电望远镜（FAST）是世界上最大的单碟射电望远镜，一直在高速产生非常大的数据量。这就需要一个高性能的数据管道来将大量的原始观测数据转化为科学数据产品。然而，在无线电数据处理中广泛使用的管道的现有解决方案不能有效地解决这种情况。本文提出了一种基于HDF 5格式的FAST流水线结构和几种I/O优化策略。首先，设计了工作流引擎，高效地驱动流水线中的各种任务;其次，在HDF 5格式的基础上设计了通用的无线数据存储规范，并开发了一个快速转换器，将原始的FITS格式转换为新的HDF 5格式;第三，我们采用了几种具体的策略来优化I/O操作，包括块存储、并行阅读/写、按需转储，在处理700 GB FAST数据的实验中，结果表明，在不进行其他优化的情况下，基于HDF 5的数据结构比原FITS格式的速度提高了1.7倍。如果采用块存储和并行I/O优化，整体性能可以达到原来的4.5倍。此外，由于FAST管道具有良好的扩展性和灵活性，我们的解决方案可以适用于其他射电望远镜。
The Five-hundred-meter Aperture Spherical Radio Telescope (FAST), which is the largest single-dish radio telescope in the world, has been producing a very large data volume with high speed. So it requires a high performance data pipeline to covert the huge raw observed data to science data product. However, the existing solutions of pipelines widely used in radio data processing cannot tackle this situation efficiently. The paper proposes a pipeline architecture for FAST based on HDF5 format and several I/O optimization strategies. First, we design the workflow engine driving the various tasks efficiently in the pipeline; second, we design a common radio data storage specification on the top of HDF5 format, and also developed a fast converter to map the original FITS format to the new HDF5 format; third, we apply several concrete strategies to optimize the I/O operations, including chunks storage, parallel reading/writing, on-demand dump, and stream process etc. In the experiment of processing 700 GB of FAST data, the results show that HDF5 based data structure without other optimizations was 1.7 times faster than original FITS format. If chunk storage and parallel I/O optimization are applied, the overall performance can reach 4.5 times as the original one. Moreover, due to the good expansibility and flexibility, our solution of FAST pipeline can be adapted to other radio telescopes.