Adaptive Query Compilation for Stream Processing

在过去几十年中，数据处理工作负载的需求发生了显著变化。如今，实时分析需要在无限的、不断变化的、高速的数据流上执行长时间运行的查询。常见的spe如Flink和Storm扩展执行，以实现高吞吐量和低延迟。然而，最近的研究表明，这些spe不能充分利用可用的硬件资源。首先，它们没有考虑特定的硬件资源进行优化。其次，它们没有考虑到不断变化的数据特征，这阻碍了各种自适应优化。第三，它们严重依赖于用户定义函数，由于数据序列化和转换，这带来了很高的处理开销。在这个项目中，我们希望面对这些挑战，以便在现代硬件上有效地处理复杂的流处理管道。为此，我们提出了一种新的自适应查询编译器，用于流处理技术，以根据硬件资源和不断变化的数据特征优化代码。此外，我们研究了将复杂的用户定义函数有效地嵌入到编译管道中的可能性，以支持广泛的高级分析数据处理工作负载。

Over the last decades, the requirements of data processing workloads significantly changed. Nowadays, real-time analytics requires the execution of long-running queries over unbounded, continuously changing, high-velocity data streams. Common SPEs such as Flink and Storm scale-out execution to achieve high throughput and low-latency. However, recent research revealed that these SPEs cannot fully utilize available hardware resources. First, they do not take the particular hardware resources into account for optimization. Second, they do not take changing data characteristics intro account, which hinders a variety of adaptive optimizations. Third, they rely heavily on user-defined functions, which introduce a high processing overhead due to data serialization and transformation. In this project, we want to face these challenges to enable efficient processing of complex stream processing pipelines on modern hardware. To this end, we propose a novel adaptive query compiler for stream processing techniques to optimize code with regards to the hardware resources and changing data characteristics. Furthermore, we study possibilities to embed complex user-defined functions into compiled pipelines efficiently to support a wide range of advanced analytical data processing workloads.