Buttressing volatile desktop grids with cloud resources within a reconfigurable environment service for workflow orchestration
Buttressing volatile desktop grids with cloud resources within a reconfigurable environment service for workflow orchestration
复制标题
DOI：
10.1186/2192-113x-3-1
复制
发表时间：
2014-01-01
影响因子：
4
通讯作者：
Kacsuk, Peter
中科院分区：
文献类型：
--
作者：
Winter, Stephen C.;Reynolds, Christopher J.;Kacsuk, Peter
云技术有可能扩大对电子科学研究的高性能计算资源的访问，但对许多科学家来说，参与这项技术的障碍仍然很高。工作流通过隐藏底层计算基础设施的细节来帮助克服障碍，并且可以在包括云在内的各种平台之间进行移植;它们也越来越多地被电子科学研究社区所接受。工作流程系统的范围和工作流程开发的复杂性所产生的问题已经得到解决，重点是工作流程的互操作性，并为不同的科学界提供定制的支持。然而，这种环境的部署可能具有挑战性，即使在用户要求相对温和的情况下。RESWO（可重新配置的工作流演示环境服务）是一种虚拟平台即服务云模型，允许在云中组装和部署更精简的定制环境。合适的分布式计算资源并不总是容易负担得起的，并且可能对科学家的参与构成进一步的障碍。使用组织内可用的空闲CPU周期的桌面网格对许多人来说是一种具有吸引力的廉价基础设施类型，并且已经有效地虚拟化为基于云的资源。然而，这种环境中的主机是不稳定的：导致尾部问题，其中一些任务随机延迟，影响整体性能。为了解决这个问题，已经开发了新的算法来实现云突发调度器，其中持久的基于云的CPU资源可以执行已经被延迟的作业的副本。本文介绍了一个RESWO实例的开发经验，其中一个桌面网格与云中的CPU资源相结合，以支持生物科学研究人员的愿望。该架构的核心组件CloudBursting调度器实现了一种算法来执行后期作业检测、云资源管理和作业监控。所获得的实验结果表明，在生物科学研究中的用例所示的显着的性能改进和好处。
Cloud technology has the potential for widening access to high-performance computational resources for e-science research, but barriers to engagement with the technology remain high for many scientists. Workflows help overcome barriers by hiding details of underlying computational infrastructure and are portable between various platforms including cloud; they are also increasingly accepted within e-science research communities. Issues arising from the range of workflow systems available and the complexity of workflow development have been addressed by focusing on workflow interoperability, and providing customised support for different science communities. However, the deployments of such environments can be challenging, even where user requirements are comparatively modest. RESWO (Reconfigurable Environment Service for Workflow Orchestration) is a virtual platform-as-a-service cloud model that allows leaner customised environments to be assembled and deployed within a cloud. Suitable distributed computation resources are not always easily affordable and can present a further barrier to engagement by scientists. Desktop grids that use the spare CPU cycles available within an organisation are an attractively inexpensive type of infrastructure for many, and have been effectively virtualised as a cloud-based resource. However, hosts in this environment are volatile: leading to the tail problem, where some tasks become randomly delayed, affecting overall performance. To solve this problem, new algorithms have been developed to implement a cloudbursting scheduler in which durable cloud-based CPU resources may execute replicas of jobs that have become delayed. This paper describes experiences in the development of a RESWO instance in which a desktop grid is buttressed with CPU resources in the cloud to support the aspirations of bioscience researchers. A core component of the architecture, the cloudbursting scheduler, implements an algorithm to perform late job detection, cloud resource management and job monitoring. The experimental results obtained demonstrate significant performance improvements and benefits illustrated by use cases in bioscience research.