Selecting resources for distributed dataflow systems according to runtime targets
Selecting resources for distributed dataflow systems according to runtime targets
复制标题
根据运行时目标为分布式数据流系统选择资源
DOI：
10.1109/pccc.2016.7820629
复制
发表时间：
2016
期刊：
影响因子：
--
通讯作者：
O. Kao
中科院分区：
文献类型：
--
作者：
L. Thamsen;I. Verbitskiy;Florian Schmidt;T. Renner;O. Kao
Distributed dataflow systems like Spark or Flink enable users to analyze large datasets. Users create programs by providing sequential user-defined functions for a set of well-defined operations, select a set of resources, and the systems automatically distribute the jobs across these resources. However, selecting resources for specific performance needs is inherently difficult and users consequently tend to overprovision, which results in poor cluster utilization. At the same time, many important jobs are executed recurringly in production clusters. This paper presents Bell, a practical system that monitors job execution, models the scale-out behavior of jobs based on previous runs, and selects resources according to user-provided runtime targets. Bell automatically chooses between different runtime prediction models to optimally support different distributed dataflow systems. Bell is implemented as a job submission tool for YARN and, thus, works with existing cluster setups. We evaluated Bell's runtime prediction with six exemplary data analytics jobs using both Spark and Flink. We present the learned scale-out models for these jobs and evaluate the relative prediction error using cross-validation, showing that our model selection approach provides better overall performance than the individual prediction models.
影响因子：
4.2
作者：
Alexandrov, Alexander;Bergmann, Rico;Warneke, Daniel
通讯作者：
Warneke, Daniel
DOI：
10.1145/1807128.1807148
发表时间：
2010-06
期刊：
--
影响因子：
--
作者：
Dominic Battré;Stephan Ewen;Fabian Hueske;O. Kao;V. Markl;Daniel Warneke
通讯作者：
Dominic Battré;Stephan Ewen;Fabian Hueske;O. Kao;V. Markl;Daniel Warneke
DOI：
10.14778/2350229.2350245
发表时间：
2012-07
期刊：
Proc. VLDB Endow.
影响因子：
--
作者：
Stephan Ewen;K. Tzoumas;Moritz Kaufmann;V. Markl
通讯作者：
Stephan Ewen;K. Tzoumas;Moritz Kaufmann;V. Markl