Serverless execution of scientific workflows: Experiments with HyperFlow, AWS Lambda and Google Cloud Functions
Serverless execution of scientific workflows: Experiments with HyperFlow, AWS Lambda and Google Cloud Functions
复制标题
DOI：
10.1016/j.future.2017.10.029
复制
发表时间：
2017-11
期刊：
影响因子：
--
通讯作者：
M. Malawski;A. Gajek;Adam Zima;B. Baliś;Kamil Figiela
中科院分区：
文献类型：
--
作者：
M. Malawski;A. Gajek;Adam Zima;B. Baliś;Kamil Figiela
由大量相互依赖的任务组成的科学工作流程代表了一类重要的复杂科学应用程序。最近，出现了一种新型的无服务器基础设施，以 Google Cloud Functions 和 AWS Lambda 等服务为代表，也称为函数即服务模型。在本文中，我们将研究此类无服务器基础设施，这些基础设施主要用于处理 Web 和物联网应用程序的后台任务，或事件驱动的流处理。我们评估它们对计算和数据密集型科学工作流程的适用性，并讨论重新利用无服务器架构来执行科学工作流程的可能方法。我们使用 AWS Lambda 和 Google Cloud Functions 以及 HyperFlow 工作流引擎开发了原型工作流执行器函数。这些函数可以在 AWS 和 Google 基础设施中运行工作流任务，并具有将数据暂存到 S3 或 Google Cloud Storage 或从 S3 或 Google Cloud Storage 暂存以及执行自定义应用程序二进制文件等功能。我们已经成功部署并执行了通常用作基准的蒙太奇天文学工作流程，并且我们报告了其性能评估的初步结果。我们的研究结果表明，简单的操作模式使这种方法易于使用，尽管准备可移植应用程序二进制文件以在远程环境中执行会产生成本。虽然我们的解决方案是早期原型，但我们发现所提出的方法非常有前途。我们还讨论了与在无服务器基础设施中执行科学工作流程相关的未来可能采取的步骤。最后，我们进行成本分析并讨论一般科学应用的资源管理的影响。
Scientific workflows consisting of a high number of interdependent tasks represent an important class of complex scientific applications. Recently, a new type ofserverlessinfrastructures has emerged, represented by such services as Google Cloud Functions and AWS Lambda, also referred to as the Function-as-a-Service model. In this paper we take a look at such serverless infrastructures, which are designed mainly for processing background tasks of Web and Internet of Things applications, or event-driven stream processing. We evaluate their applicability to more compute- and data-intensive scientific workflows and discuss possible ways to repurpose serverless architectures for execution of scientific workflows. We have developed prototype workflow executor functions using AWS Lambda and Google Cloud Functions, coupled with the HyperFlow workflow engine. These functions can run workflow tasks in AWS and Google infrastructures, and feature such capabilities as data staging to/from S3 or Google Cloud Storage and execution of custom application binaries. We have successfully deployed and executed the Montage astronomy workflow, often used as a benchmark, and we report on initial results of its performance evaluation. Our findings indicate that the simple mode of operation makes this approach easy to use, although there are costs involved in preparing portable application binaries for execution in a remote environment.While our solution is an early prototype, we find the presented approach highly promising. We also discuss possible future steps related to execution of scientific workflows in serverless infrastructures. Finally, we perform a cost analysis and discuss implications with regard to resource management for scientific applications in general.