Meteor: Optimizing spark-on-yarn for short applications
Meteor: Optimizing spark-on-yarn for short applications
复制标题
DOI：
10.1016/j.future.2019.05.077
复制
发表时间：
2019-12
期刊：
影响因子：
--
通讯作者：
Hong Zhang;Hai Huang;Liqiang Wang
中科院分区：
文献类型：
--
作者：
Hong Zhang;Hai Huang;Liqiang Wang
由于其速度和易用性，Spark已成为数据科学家分析各种大小数据的流行工具。与直觉相反，工业公司（如Google，Facebook和Yahoo）的数据处理工作负载主要是短期运行的应用程序，这是由于大多数应用程序主要由简单的SQL类查询组成（Dean，2004，Zaharia等人，2008）。不幸的是，Spark的当前版本并没有针对这类工作负载进行优化。在本文中，我们提出了一种新的框架，称为流星，它可以显着提高性能的短期运行的应用程序。我们为Spark扩展了三种额外的操作模式：单线程、单容器和分布式。单线程模式只在一个线程上执行所有任务;单容器模式通过多线程在一个容器中运行这些任务;分布式模式将所有任务分配给整个集群。还设计了一个新的提交应用程序的框架，该框架利用细粒度的Spark性能模型来决定在新的应用程序提交时调用三种模式中的哪一种是最有效的。从我们在Amazon EC2上的大量实验来看，当输入大小较小时，单线程模式是最佳选择，否则分布式模式更好。总体而言，Meteor在短应用程序中比原始Spark快2倍。
Due to its speed and ease of use, Spark has become a popular tool amongst data scientists to analyze data in various sizes. Counter-intuitively, data processing workloads in industrial companies such as Google, Facebook, and Yahoo are dominated by short-running applications, which is due to the majority of applications being mostly consisted of simple SQL-like queries (Dean, 2004, Zaharia et al, 2008). Unfortunately, the current version of Spark is not optimized for such kinds of workloads. In this paper, we propose a novel framework, called Meteor, which can dramatically improve the performance for short-running applications. We extend Spark with three additional operating modes: one-thread, one-container, and distributed. The one-thread mode executes all tasks on just one thread; the one-container mode runs these tasks in one container by multi-threading; the distributed mode allocates all tasks over the whole cluster. A new framework for submitting applications is also designed, which utilizes a fine-grained Spark performance model to decide which of the three modes is the most efficient to invoke upon a new application submission. From our extensive experiments on Amazon EC2, one-thread mode is the optimal choice when the input size is small, otherwise the distributed mode is better. Overall, Meteor is up to 2 times faster than the original Spark for short applications.