In Search of a Fast and Efficient Serverless DAG Engine
In Search of a Fast and Efficient Serverless DAG Engine
复制标题
DOI：
10.1109/pdsw49588.2019.00005
复制
发表时间：
2019-10
期刊：
影响因子：
--
通讯作者：
Benjamin Carver;Jingyuan Zhang;Ao Wang;Yue Cheng
中科院分区：
文献类型：
--
作者：
Benjamin Carver;Jingyuan Zhang;Ao Wang;Yue Cheng
Python编写的数据分析应用程序可以建模为基于有向无环图（DAG）的工作流，并编译为基于DAG的工作流，其中节点是细粒度任务，边是任务依赖关系。此类分析工作流作业越来越多地以具有大扇出的短细粒度任务为特征。这些特性使它们非常适合称为无服务器计算或功能即服务（FaaS）的新云计算模型，近年来已变得流行。无服务器计算平台的自动扩展特性可以适应短任务和突发工作负载，而无服务器计算提供商的按使用付费计费模式可以降低短任务的成本。在本文中，我们深入研究了无服务器计算中的DAG调度问题空间。我们确定并评估了一组技术，使DAG服务器感知。这些技术已在WUKONG中实现，WUKONG是一个与AWS Lambda协调的无服务器DAG调度器。WUKONG通过静态和动态调度的组合提供分散调度。我们提出了一个实证研究的结果，其中悟空是适用于一系列的微基准和现实世界的DAG应用程序。结果证明了WUKONG在最大限度地减少AWS Lambda引入的性能开销方面的有效性-与服务器DAG调度程序相比，WUKONG实现了具有竞争力的性能，同时在更大规模下将实际DAG作业的性能提高了4.1倍。
Python-written data analytics applications can be modeled as and compiled into a directed acyclic graph (DAG) based workflow, where the nodes are fine-grained tasks and the edges are task dependencies.Such analytics workflow jobs are increasingly characterized by short, fine-grained tasks with large fan-outs. These characteristics make them well-suited for a new cloud computing model called serverless computing or Function-as-a-Service (FaaS), which has become prevalent in recent years. The auto-scaling property of serverless computing platforms accommodates short tasks and bursty workloads, while the pay-per-use billing model of serverless computing providers keeps the cost of short tasks low. In this paper, we thoroughly investigate the problem space of DAG scheduling in serverless computing. We identify and evaluate a set of techniques to make DAG schedulers serverless-aware. These techniques have been implemented in WUKONG , a serverless, DAG scheduler attuned to AWS Lambda. WUKONG provides decentralized scheduling through a combination of static and dynamic scheduling. We present the results of an empirical study in which WUKONG is applied to a range of microbenchmark and real-world DAG applications. Results demonstrate the efficacy of WUKONG in minimizing the performance overhead introduced by AWS Lambda — WUKONG achieves competitive performance compared to a serverful DAG scheduler, while improving the performance of real-world DAG jobs by as much as 4.1x at larger scale.