Adaptive Query Compilation for Stream Processing

流处理的自适应查询编译

• 批准号: 447268056
• 负责人: Professor Dr. Volker Markl
• 金额: --
• 依托单位: Fachgebiet Datenbanksysteme und Informationsmanagement (DIMA)
• 依托单位国家: 德国
• 项目类别: Priority Programmes
• 资助国家: 德国
• 项目状态: 未结题
• 来源: https://gepris.dfg.de/gepris/projekt/447268056?language=en
• 关键词: Adaptive; Query; Compilation; Stream; Processing

Over the last decades, the requirements of data processing workloads significantly changed. Nowadays, real-time analytics requires the execution of long-running queries over unbounded, continuously changing, high-velocity data streams. Common SPEs such as Flink and Storm scale-out execution to achieve high throughput and low-latency. However, recent research revealed that these SPEs cannot fully utilize available hardware resources. First, they do not take the particular hardware resources into account for optimization. Second, they do not take changing data characteristics intro account, which hinders a variety of adaptive optimizations. Third, they rely heavily on user-defined functions, which introduce a high processing overhead due to data serialization and transformation. In this project, we want to face these challenges to enable efficient processing of complex stream processing pipelines on modern hardware. To this end, we propose a novel adaptive query compiler for stream processing techniques to optimize code with regards to the hardware resources and changing data characteristics. Furthermore, we study possibilities to embed complex user-defined functions into compiled pipelines efficiently to support a wide range of advanced analytical data processing workloads.

在过去的几十年中，数据处理工作负载的要求发生了显著变化。如今，实时分析需要在无限、不断变化的高速数据流上执行长时间运行的查询。Flink和Storm等常见SPE横向扩展执行，以实现高吞吐量和低延迟。然而，最近的研究表明，这些SPE无法充分利用可用的硬件资源。首先，它们不考虑特定的硬件资源进行优化。其次，它们没有考虑不断变化的数据特征，这阻碍了各种自适应优化。第三，它们严重依赖于用户定义的函数，由于数据序列化和转换，这会引入很高的处理开销。在这个项目中，我们希望面对这些挑战，以便在现代硬件上有效处理复杂的流处理管道。为此，我们提出了一种新的自适应查询编译器的流处理技术，以优化代码的硬件资源和不断变化的数据特性。此外，我们研究了将复杂的用户定义函数有效地嵌入到编译管道中的可能性，以支持各种高级分析数据处理工作负载。