mcatCS: A Highly Efficient Cross-matching Scheme for Multi-band Astronomical Catalogs
mcatCS: A Highly Efficient Cross-matching Scheme for Multi-band Astronomical Catalogs
复制标题
mcatCS：多波段天文目录的高效交叉匹配方案
DOI：
10.1088/1538-3873/ab024c
复制
发表时间：
2019
影响因子：
3.5
通讯作者：
Fan Dongwei
中科院分区：
文献类型：
--
作者：
Li Bingyao;Yu Ce;Li Chen;Hu Xiaoteng;Xiao Tian;Tang Shanjiang;Cui Chenzhou;Fan Dongwei
多波段天文目录交叉匹配一直是，并将继续是天文学研究不可或缺的。然而，不同波段的存档数据量非常巨大，这导致交叉匹配过程具有高计算消耗和响应缓慢。观测数据的不断增长也将增加复杂性。在本文中，我们提出了mcatCS（多波段目录交叉匹配方案），一个分布式的交叉匹配方案，以有效地整合天体数据从十亿行多波段天文目录。它部署在商用机器集群上，并为最终用户提供基于命令行的界面。为了实现快速交叉匹配，星表中的数据被重新格式化为分组空间索引文件，这是一种专门设计的多波段星表统一格式。此外，最小冲突的数据布局策略，以最大限度地提高交叉匹配的并行化。利用中国国家天文台的真实的数据，验证了mcatCS具有良好的十亿行多波段星表交叉匹配能力，实验结果表明，其查询响应速度比MongoDB提高38%~ 45%，比使用HEALPix B树索引的PostgreSQL提高21%~ 32%.此外，虽然Q3 C和H3C-PostgreSQL的扩展索引包-提供了更快的查询响应速度小于8500万个源，mcatCS被证明是有利的源扩展到1亿后，并实现了时间减少30.3%和30.7%相比，Q3 C和H3C为2亿个源。
Multi-band astronomical catalog cross-matching has always been, and will continue to be, indispensable to astronomy research. However, the archived data volume in different wavebands is extremely huge, which results in the cross-matching process having high computational consumption and slow response. The complexity will also be augmented by the continuous growth of observational data. In this paper, we present mcatCS (multi-band catalog Cross-matching Scheme), a distributed cross-matching scheme to efficiently integrate celestial object data from billion-row multi-band astronomical catalogs. It is deployed on a cluster of commodity machines and provides a command-line-based interface to the end user. To allow fast cross-matching, the data in catalogs are reformatted into the Grouped Spatial Index File, which is a specially designed multi-band catalog uniform format. Furthermore, a min-conflicts data layout strategy is utilized to maximize the parallelization of cross-matching. Using real data, archived in the National Astronomical Observatories of China, we verify that mcatCS has good capabilities for performing efficient and reliable cross-matching between billion-row multi-band catalogs, and experimental results show that the query response speed is 38% to 45% greater than that of MongoDB and 21% to 32% greater than that of PostgreSQL with the HEALPix B-tree index. Moreover, although Q3C and H3C—the extension index packages for PostgreSQL—offer faster query response speed for less than 85 million sources, mcatCS proves to be advantageous after sources scale up to 100 million, and achieves a time reduction of 30.3% and 30.7% compared to Q3C and H3C for 200 million sources.