Monotasks: Architecting for Performance Clarity in Data Analytics Frameworks
Monotasks: Architecting for Performance Clarity in Data Analytics Frameworks
复制标题
单任务：数据分析框架中性能清晰度的架构
DOI：
10.1145/3132747.3132766
复制
发表时间：
2017
期刊：
影响因子：
--
通讯作者：
Shenker, Scott
中科院分区：
文献类型：
--
作者：
Ousterhout, Kay;Canel, Christopher;Ratnasamy, Sylvia;Shenker, Scott
在当今的数据分析框架中，许多用户很难对其工作负载的性能进行推理。在不了解哪些因素对性能最重要的情况下，用户无法确定要设置哪些配置参数以及使用哪些硬件来优化运行时。本文探索了一个系统体系结构，旨在使用户能够更容易地对性能瓶颈进行推理。我们建议将作业分解为单任务：每个工作单元都使用一种资源，而不是像今天的框架中那样，将作业分解为输送许多资源的任务。我们证明了显式地分离不同资源的使用简化了关于性能的推理，而不会牺牲性能。对于典型的场景，单任务提供的作业完成时间不超过ApacheSpark的9%，并导致了一个作业完成时间模型，该模型预测不同硬件和软件配置下的运行时间，误差最高为28%。此外，将不同资源的使用分开允许进行新的优化以提高性能。
In today's data analytics frameworks, many users struggle to reason about the performance of their workloads. Without an understanding of what factors are most important to performance, users can't determine what configuration parameters to set and what hardware to use to optimize runtime. This paper explores a system architecture designed to make it easy for users to reason about performance bottlenecks. Rather than breaking jobs into tasks that pipeline many resources, as in today's frameworks, we propose breaking jobs into monotasks: units of work that each use a single resource. We demonstrate that explicitly separating the use of different resources simplifies reasoning about performance without sacrificing performance. Monotasks provide job completion times within 9% of Apache Spark for typical scenarios, and lead to a model for job completion time that predicts runtime under different hardware and software configurations with at most 28% error. Furthermore, separating the use of different resources allows for new optimizations to improve performance.