Streaming Augmented Lineage: Traceability of Complex Stream Data Analysis
Streaming Augmented Lineage: Traceability of Complex Stream Data Analysis
复制标题
流式增强谱系：复杂流数据分析的可追溯性
DOI：
10.1007/978-3-031-21047-1_20
复制
发表时间：
2022
期刊：
影响因子：
--
通讯作者：
Akiyoshi Matono
中科院分区：
文献类型：
--
作者：
Masaya Yamada;Hiroyuki Kitagawa;Salman Ahmed Shaikh;Toshiyuki Amagasa;Akiyoshi Matono
数据沿袭指定了哪些源元组对哪些输出元组有贡献，并使我们能够确保分析的可追溯性。如今，无数的物联网设备和传感器正在高速产生大量的流数据。流处理通常被部署为实时地连续处理这样的流数据。由于人工智能和机器学习的快速发展，现代数据分析变得更加复杂，流处理也不例外。需要丰富数据沿袭，以解决日益增加的复杂性。在本文中，我们提出了流增强血统。增强血统介绍了AI/ML处理沿着普通血统导出输出数据的原因，以增强包括AI/ML处理在内的复杂数据分析的可追溯性。流增强沿袭在复杂流处理中提供增强沿袭。我们描述了如何在Flink上实现我们的方案，并对原型系统进行了详细的性能评估。
Data lineage specifies which source tuples contribute to which output tuples and enables us to ensure the traceability of the analysis. Today, innumerable IoT devices and sensors are producing massive amounts of stream data at high velocity. Stream processing is often deployed to continuously process such stream data in realtime. Modern data analysis has become more complex due to the rapid development of AI and machine learning, and stream processing is no exception. Data lineage needs to be enriched to address this increasing complexity. In this paper, we proposestreaming augmented lineage.Augmented lineagepresents the reason why AI/ML processing derives the output data along with the ordinary lineage to enhance the traceability of the complex data analysis including AI/ML processing. Streaming augmented lineage provides augmented lineage in complex stream processing. We describe how to implement our scheme on Flink, and show detailed performance evaluations on the prototype system.