Compiling and Optimizing Iterative Data Analysis Programs with Shared State on Evolving Datasets

在不断变化的数据集上编译和优化具有共享状态的迭代数据分析程序

• 批准号: 248356729
• 负责人: Professor Dr. Volker Markl
• 金额: --
• 依托单位: Fachgebiet Datenbanksysteme und Informationsmanagement (DIMA)
• 依托单位国家: 德国
• 项目类别: Research Units
• 财政年份: 2013
• 资助国家: 德国
• 起止时间: 2012-12-31 至 2016-12-31
• 项目状态: 已结题
• 来源: https://gepris.dfg.de/gepris/projekt/248356729?language=en
• 关键词: Compiling; Optimizing; Iterative; Data; Analysis

The goal of Project A within the Stratosphere II research unit is to research, design, and develop a data programming language, an associated optimizing compiler, an intermediate data- and control- flow representation, and an optimizer that determines efficient execution strategies for workloads of data analysis programs with iterations and stateful operators, both over static data and over infinite, evolving datasets. The project will research language abstractions for specifying iterations and state, novel optimizations for iterative and stateful programs targeting performance as well as novel fault tolerance schemes for massively parallel iterative algorithms, and optimization of workloads of programs including work sharing between programs. The project will also demonstrate the overall effectiveness of Stratosphere II by integrating the results of all projects in a coherent system, identifying a relevant use-case workload, and evaluating and benchmarking the system performance. In particular, this project aims at answering following questions:1. What are the necessary language and system primitives to abstract parallelization and state, and expose mutable state management to the programmer of DAPs without compromizing scalability, performance, and fault tolerance?2. What are the optimizing program transformations that apply to a DAP with state and iterations and create a more efficient program?3. To what extent can an optimizing language compiler optimize advanced data analytics applications with state and iterations?4. To what extent can we support the fault-tolerant and efficient execution of DAPs with state via languagelevel features that expose algorithmic aspects of programs?5. How can we build an optimizer for workloads of DAPs to optimize state management across DAPs?

Stratosphere II研究单元内的项目A的目标是研究、设计和开发一种数据编程语言、一个相关的优化编译器、一个中间数据流和控制流表示以及一个优化器，该优化器为具有迭代和有状态运算符的数据分析程序的工作负载确定有效的执行策略，无论是在静态数据上还是在无限的、不断发展的数据集上。该项目将研究用于指定迭代和状态的语言抽象，针对性能的迭代和有状态程序的新颖优化，以及大规模并行迭代算法的新颖容错方案，以及程序工作负载的优化，包括程序之间的工作共享。该项目还将通过将所有项目的成果整合到一个连贯的系统中，确定相关的用例工作量，并对系统性能进行评估和基准测试，来展示平流层II的总体有效性。具体而言，本项目旨在回答以下问题：1。什么是必要的语言和系统原语来抽象并行化和状态，并向DAP的程序员公开可变的状态管理，而不会损害可伸缩性，性能和容错性？2.什么是应用于具有状态和迭代的DAP并创建更有效程序的优化程序转换？3.优化语言编译器可以在多大程度上优化具有状态和迭代的高级数据分析应用程序？4.在多大程度上，我们可以通过暴露程序算法方面的语言级功能来支持具有状态的DAP的容错和有效执行？5.我们如何为DAP的工作负载构建优化器，以优化DAP之间的状态管理？

项目成果

BlockJoin: Efficient Matrix Partitioning Through Joins

BlockJoin：通过连接进行高效的矩阵分区

• DOI: 10.14778/3151106.3151110
• 发表时间: 2017
• 期刊: Proc. VLDB Endow.
• 作者: Andreas

"All roads lead to Rome": optimistic recovery for distributed iterative data processing

• DOI: 10.1145/2505515.2505753
• 发表时间: 2013-10
• 期刊: Proceedings of the 22nd ACM international conference on Information & Knowledge Management
• 作者: Sebastian Schelter;Stephan Ewen;K. Tzoumas;V. Markl

Spinning Fast Iterative Data Flows

• DOI: 10.14778/2350229.2350245
• 发表时间: 2012-07
• 期刊: Proc. VLDB Endow.
• 作者: Stephan Ewen;K. Tzoumas;Moritz Kaufmann;V. Markl

Emma in Action: Declarative Dataflows for Scalable Data Analysis

• DOI: 10.1145/2882903.2899396
• 发表时间: 2016-06
• 期刊: Proceedings of the 2016 International Conference on Management of Data
• 作者: Alexander B. Alexandrov;Andreas Salzmann;Georgi Krastev;Asterios Katsifodimos;V. Markl

Massively parallel data analysis with PACTs on Nephele

• DOI: 10.14778/1920841.1921056
• 发表时间: 2010-09
• 期刊: Proceedings of the VLDB Endowment
• 影响因子: 2.5
• 作者: Alexander B. Alexandrov;Dominic Battré;Stephan Ewen;Max Heimel;Fabian Hueske;O. Kao;V. Markl;Erik Nijkamp;Daniel Warneke