Transparent Orchestration of Task-based Parallel Applications in Containers Platforms
Transparent Orchestration of Task-based Parallel Applications in Containers Platforms
复制标题
容器平台中基于任务的并行应用程序的透明编排
DOI：
10.1007/s10723-017-9425-z
复制
发表时间：
2018
影响因子：
5.5
通讯作者：
Rosa M. Badia
中科院分区：
文献类型：
--
作者：
Cristian Ramon;Albert Serven;J. Ejarque;D. Lezzi;Rosa M. Badia
本文提出了一个框架，以方便地建立和执行并行应用程序在基于容器的分布式计算平台上，在用户透明的方式。所提出的框架是COMP Superscalar（COMPPS）编程模型和运行时的组合，它提供了一种从顺序代码开发基于任务的并行应用程序的简单方法，以及简化计算环境中应用程序部署的容器管理平台（如Docker，Mesos或Singularity）。该框架为科学家和开发人员提供了一种简单的方法来实现并行分布式应用程序，并以一键方式部署它们。我们已经构建了一个原型，它在不同的场景中将COMPS与不同的容器引擎集成在一起：i）Docker集群，ii）Mesos集群，iii）HPC集群中的Singularity。我们已经评估了两个基准应用程序的构建阶段、部署和执行的开销，并将其与基于KVM和OpenStack的云测试床以及使用裸金属节点进行了比较。在构建和部署阶段，与云环境相比，我们观察到了一个重要的收益。这使得能够相对于计算负载更好地适配资源。相比之下，我们在执行过程中发现了额外的开销，这主要是由于多主机Docker网络。
This paper presents a framework to easily build and execute parallel applications in container-based distributed computing platforms in a user-transparent way. The proposed framework is a combination of the COMP Superscalar (COMPSs) programming model and runtime, which provides a straightforward way to develop task-based parallel applications from sequential codes, and containers management platforms that ease the deployment of applications in computing environments (as Docker, Mesos or Singularity). This framework provides scientists and developers with an easy way to implement parallel distributed applications and deploy them in a one-click fashion. We have built a prototype which integrates COMPSs with different containers engines in different scenarios: i) a Docker cluster, ii) a Mesos cluster, and iii) Singularity in an HPC cluster. We have evaluated the overhead in the building phase, deployment and execution of two benchmark applications compared to a Cloud testbed based on KVM and OpenStack and to the usage of bare metal nodes. We have observed an important gain in comparison to cloud environments during the building and deployment phases. This enables better adaptation of resources with respect to the computational load. In contrast, we detected an extra overhead during the execution, which is mainly due to the multi-host Docker networking.