Mapping the Big Data Landscape: Technologies, Platforms and Paradigms for Real-Time Analytics of Data Streams
Mapping the Big Data Landscape: Technologies, Platforms and Paradigms for Real-Time Analytics of Data Streams
复制标题
DOI：
10.1109/access.2020.3046132
复制
发表时间：
2021
期刊：
影响因子：
3.9
通讯作者：
T. Dubuc;Frederic T. Stahl;E. Roesch
中科院分区：
文献类型：
--
作者：
T. Dubuc;Frederic T. Stahl;E. Roesch
昨天的“大数据”就是今天的“数据”。随着技术的进步，新的挑战出现，新的解决方案被开发出来。由于物联网应用在过去十年中的出现，数据挖掘领域一直面临着实时处理和分析数据流的挑战，并且在高数据吞吐量条件下。这通常被称为大数据的速度方面。虽然有许多评论数据流挖掘技术和应用程序，有很少的工作调查数据流处理的范例和相关技术，从数据收集到预处理和功能处理，从用户的角度来看，而不是服务提供商。在本文中，我们评估了一种特定类型的解决方案，该解决方案侧重于流数据和处理管道，这些管道允许在线分析无法在计算平台上按原样存储的数据流。我们回顾基本的计算概念，如分布式计算，容错计算和计算范式/架构。然后，我们回顾了可用的技术解决方案，以及与数据流挖掘相关的应用程序，作为这些理论概念的案例研究。最后，我们讨论了数据流处理/分析领域，未来的发展方向和研究挑战。
The ‘Big Data’ of yesterday is the ‘data’ of today. As technology progresses, new challenges arise and new solutions are developed. Due to the emergence of Internet of Things applications within the last decade, the field of Data Mining has been faced with the challenge of processing and analysing data streams in real-time, and under high data throughput conditions. This is often referred to as the Velocity aspect of Big Data. Whereas there are numerous reviews on Data Stream Mining techniques and applications, there is very little work surveying Data Stream processing paradigms and associated technologies, from data collection through to pre-processing and feature processing, from the perspective of the user, not that of the service provider. In this article, we evaluate a particular type of solution, which focuses on streaming data, and processing pipelines that permit online analysis of data streams that cannot be stored as-is on the computing platform. We review foundational computational concepts such as distributed computation, fault-tolerant computing, and computational paradigms/architectures. We then review the available technological solutions, and applications that pertain to data stream mining as case studies of these theoretical concepts. We conclude with a discussion of the field of data stream processing/analytics, future directions and research challenges.