Optimization of Row Pattern Matching over Sequence Data in Spark SQL
Optimization of Row Pattern Matching over Sequence Data in Spark SQL
复制标题
Spark SQL 中序列数据行模式匹配的优化
DOI：
10.1007/978-3-030-27615-7_1
复制
发表时间：
2019
期刊：
影响因子：
--
通讯作者：
Nasu Yuya
中科院分区：
文献类型：
--
作者：
Nakabasami Kosuke;Kitagawa Hiroyuki;Nasu Yuya
由于信息通信技术和传感器技术的进步，每天都会产生和处理大量的序列数据（时间序列数据、日志数据等）。 2016年，存储在关系数据库中的序列数据的行模式匹配被标准化为SQL/RPR。如今，除了关系数据库之外，还有许多用于在并行和分布式计算环境中处理大量数据的框架。它们包括 MapReduce 和 Spark。 Hive 和 Spark SQL 使我们能够使用类似 SQL 的查询语言编写数据分析过程。行模式匹配在 Hive 和 Spark SQL 中也很有用。然而，行模式匹配过程的计算成本很大并且需要使该过程高效。在本文中，我们提出了两种优化方法来实现行模式匹配过程计算成本的降低。我们重点关注 Spark，并展示 Spark SQL 所提出方法的设计和实现。我们通过实验验证了我们的优化方法确实有助于减少 Spark SQL 查询（包括行模式匹配）的处理时间。
Due to the advance of information and communications technology and sensor technology, a large quantity of sequence data (time series data, log data, etc.) are generated and processed every day. Row pattern matching for the sequence data stored in relational databases was standardized as SQL/RPR in 2016. Today, in addition to relational databases, there are many frameworks for processing a large amount of data in parallel and distributed computing environments. They include MapReduce and Spark. Hive and Spark SQL enable us to code data analysis processes in SQL-like query languages. Row pattern matching is also beneficial in Hive and Spark SQL. However, computational cost of the row pattern matching process is large and it is needed to make this process efficient. In this paper, we propose two optimization methods to realize the reduction of computational cost for row pattern matching process. We focus on Spark and show design and implementation of the proposed methods for Spark SQL. We verify by the experiments that our optimization methods really contribute to the reduction of the processing time of Spark SQL queries including row pattern matching.