BESIII Physical Analysis on Hadoop Platform
BESIII Physical Analysis on Hadoop Platform
复制标题
Hadoop 平台上的 BESIII 物理分析
DOI：
10.1088/1742-6596/513/3/032044
复制
发表时间：
2014
期刊：
影响因子：
--
通讯作者：
孙功星
中科院分区：
文献类型：
--
作者：
孙功星
在过去的20年里，计算集群被广泛地应用于高能物理数据处理。运行在传统数据到计算结构的集群上的作业，不得不通过网络读取大量数据到计算节点进行分析，从而使I/O延迟成为整个系统的瓶颈。基于MapReduce编程模型的新型分布式计算技术具有高并发性、高可伸缩性和高容错性等优点，在处理大数据方面可以有所裨益。提出了利用MapReduce模型进行BESIII物理分析的思想，提出了一种基于Hadoop平台的数据分析系统结构，不仅大大提高了数据分析的效率，而且降低了系统建设的成本。此外，本文还建立了基于事件级元数据(标签库)的事件预选系统，以优化数据分析流程。
In the past 20 years, computing cluster has been widely used for High Energy Physics data processing. The jobs running on the traditional cluster with a Data-to-Computing structure, have to read large volumes of data via the network to the computing nodes for analysis, thereby making the I/O latency become a bottleneck of the whole system. The new distributed computing technology based on the MapReduce programming model has many advantages, such as high concurrency, high scalability and high fault tolerance, and it can benefit us in dealing with Big Data. This paper brings the idea of using MapReduce model to do BESIII physical analysis, and presents a new data analysis system structure based on Hadoop platform, which not only greatly improve the efficiency of data analysis, but also reduces the cost of system building. Moreover, this paper establishes an event pre-selection system based on the event level metadata (TAGs) database to optimize the data analyzing procedure.