jade: An End-To-End Data Transfer and Catalog Tool
jade: An End-To-End Data Transfer and Catalog Tool
复制标题
jade：端到端数据传输和编目工具
DOI：
10.1088/1742-6596/898/6/062050
复制
发表时间：
2017
期刊：
影响因子：
--
通讯作者：
P. Meade
中科院分区：
文献类型：
--
作者：
P. Meade
冰立方中微子观测站是位于地理南极的立方公里中微子望远镜。 IceCube 每天收集 1 TB 的数据。在线过滤农场实时处理这些数据，并选择 10% 通过卫星发送到威斯康星大学麦迪逊分校的主数据中心。 IceCube有两名常年现场操作员。由于南极过冬条件恶劣，每年都会雇佣新的操作员。这些操作员的任务是在严格的隔离条件下运行复杂的探测器的日常操作。他们运行的系统之一是数据归档和传输系统。由于这些具有挑战性的操作条件，数据归档和传输系统首先必须简单且稳健。它还必须共享有限的卫星带宽资源，并收集和保存有用的元数据。 IceCube 的原始数据归档和传输软件于 2005 年编写。在生产环境中运行了几年后，决定完全重写它，以解决许多结构性缺陷。新的数据归档和传输软件 (JADE2) 已经投入生产几个月，提供了改进的性能和弹性。 JADE2 的主要目标之一是提供一个统一的系统来端到端处理 IceCube 数据：从南极的收集，一直到在北极的专用存储库中长期存档和保存。在这篇文章中，我们描述了在 IceCube 等极端操作条件下的粒子物理实验中开发和操作数据存档和传输软件的经验和教训。
The IceCube Neutrino Observatory is a cubic kilometer neutrino telescope located at the Geographic South Pole. IceCube collects 1 TB of data every day. An online filtering farm processes this data in real time and selects 10% to be sent via satellite to the main data center at the University of Wisconsin-Madison. IceCube has two year-round on-site operators. New operators are hired every year, due to the hard conditions of wintering at the South Pole. These operators are tasked with the daily operations of running a complex detector in serious isolation conditions. One of the systems they operate is the data archiving and transfer system. Due to these challenging operational conditions, the data archive and transfer system must above all be simple and robust. It must also share the limited resource of satellite bandwidth, and collect and preserve useful metadata. The original data archive and transfer software for IceCube was written in 2005. After running in production for several years, the decision was taken to fully rewrite it, in order to address a number of structural drawbacks. The new data archive and transfer software (JADE2) has been in production for several months providing improved performance and resiliency. One of the main goals for JADE2 is to provide a unified system that handles the IceCube data end-to-end: from collection at the South Pole, all the way to long-term archive and preservation in dedicated repositories at the North. In this contribution, we describe our experiences and lessons learned from developing and operating the data archive and transfer software for a particle physics experiment in extreme operational conditions like IceCube.