MetricQ: A Scalable Infrastructure for Processing High-Resolution Time Series Data
MetricQ: A Scalable Infrastructure for Processing High-Resolution Time Series Data
复制标题
MetricQ：用于处理高分辨率时间序列数据的可扩展基础设施
DOI：
--
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
W. Nagel
中科院分区：
文献类型：
--
作者：
Thomas Ilsche;Daniel Hackenberg;R. Schöne;M. Höpfner;W. Nagel
在本文中，我们介绍了 MetricQ，一种用于收集、归档和分析传感器数据的新型基础设施。 MetricQ 的核心组件是基于高级消息队列协议的可扩展消息代理，以及专为时间序列数据设计的新开发的分层时间线聚合 (HTA) 存储概念。 HTA 在数据收集过程中需要进行适度的数据处理，存储空间开销约为 10%，从而将典型时间线请求的复杂度从 O(N) 降低到 O(1)。这使得能够以足以满足交互式用例的性能水平访问跨越数年的非常大的指标时间线和数十亿个数据点。与该领域的现有解决方案相比，不会丢弃数据中非常短的峰值等相关信息。我们演示了如何以非常高的更新率使用带有少量指标的 MetricQ（例如，用于能源效率研究），以及如何以中等更新率使用大量指标（例如，监控来自我们数据中心的电气和冷却基础设施的数据）。
In this paper we present MetricQ, a novel infrastructure for collecting, archiving, and analyzing sensor data. Core components of MetricQ are a scalable message broker based on the Advanced Message Queuing Protocol, and a newly developed Hierarchical Timeline Aggregation (HTA) storage concept that is specifically designed for timeseries data. HTA requires moderate data processing during data collection and a storage space overhead of about 10 %, and in turn reduces the complexity of typical timeline request from O(N) to O(1). This enables access to very large metric timelines spanning years and billions of data points at a performance level that is sufficient for interactive use cases. In contrast to existing solutions in this domain, no relevant information such as very short peaks in the data is discarded. We demonstrate how we use MetricQ with few metrics at very high update rates, e.g., for energy efficiency research, and for a very large number of metrics at moderate update rates, e.g., monitoring data from the electrical and cooling infrastructure of our data center.