Grosbeak: A Data Warehouse Supporting Resource-Aware Incremental Computing
Grosbeak: A Data Warehouse Supporting Resource-Aware Incremental Computing
复制标题
Grosbeak：支持资源感知增量计算的数据仓库
DOI：
10.1145/3318464.3384708
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
Jingren Zhou
中科院分区：
文献类型：
--
作者：
Zuozhi Wang;Kai Zeng;Botong Huang;Wei Chen;Xiaozong Cui;Bo Wang;J. Liu;Liya Fan;Dachuan Qu;Zhenyu Hou;Tao Guan;Chen Li;Jingren Zhou
作为推导决策支持见解的主要方法，自动化的经常性常规分析作业是现代企业数据仓库中集群资源使用的主要部分。这些经常出现的常规工作通常具有严格的时间表和由外部业务逻辑确定的截止日期，因此在集群中导致可怕的资源偏向和严重的资源过度交流。在本文中，我们介绍了Grosbeak，这是一个新颖的数据仓库，该仓库支持资源感知的增量计算以处理重复的常规作业，使资源偏斜平滑并优化资源使用情况。与传统数据仓库中的批处理处理不同，Grosbeak利用了不断摄入数据的事实。它将分析作业分为小批量，从而逐步处理逐渐可用的数据，并在群集具有免费资源时明智地安排这些小批次作业。在此演示中，我们使用现实世界分析管道展示了Grosbeak。用户可以通过注册重复查询并观察增量调度行为和平滑资源使用模式来与数据仓库进行交互。
As the primary approach to deriving decision-support insights, automated recurring routine analytic jobs account for a major part of cluster resource usages in modern enterprise data warehouses. These recurring routine jobs usually have stringent schedule and deadline determined by external business logic, and thus cause dreadful resource skew and severe resource over-provision in the cluster. In this paper, we present Grosbeak, a novel data warehouse that supports resource-aware incremental computing to process recurring routine jobs, smooths the resource skew, and optimizes the resource usage. Unlike batch processing in traditional data warehouses, Grosbeak leverages the fact that data is continuously ingested. It breaks an analysis job into small batches that incrementally process the progressively available data, and schedules these small-batch jobs intelligently when the cluster has free resources. In this demonstration, we showcase Grosbeak using real-world analysis pipelines. Users can interact with the data warehouse by registering recurring queries and observing the incremental scheduling behavior and smoothed resource usage pattern.