Optimal Scheduling of Scientific Application Workflows for Cloud-augmented Grid Infrastructures

研究科学家需要云计算来灵活地支持他们的计算需求，但也需要用户友好的工具来参与。科学研究中苛刻的计算需求已经通过网格计算得到了解决，直到最近，网格计算一直基于固定的物理广域基础设施。然而，物理网格资源可能会过载-科学家可能会发现这限制了他们的需求，在特定时刻所需的性能无法实时解决。用户友好的工具和环境需要支持网格计算在科学界。P-GRADE是一个基于网络的门户网站，由研究人员共同开发，用于设计，提交和监测网格上的工作流程。它是高度图形化的，因此对广泛的非计算专家非常有吸引力。熟悉对用户很重要。为了开发云计算的潜力，为同一社区的科学研究人员，这是非常可取的，从他们的角度来看，尽可能保持计算支持环境的连续性。多年来，P-GRADE环境一直受到用户需求和愿望的影响，并且鉴于其可用性，已经在研究科学界获得了高度的认可。无论如何，从头开始重新设计用户界面是一项非常昂贵的开发活动。服务网格和集群网格之间的互操作性是由欧盟FP 7资助的EDGeS项目的研究人员实现的，因此每个社区的用户现在可以访问另一个社区的资源。网格计算也可以扩展到云资源上。研究人员最近开发了扩展服务和集群网格与虚拟云资源的解决方案。研究人员与威斯敏斯特大学和帝国理工学院的生物研究小组合作，对为网格开发的工具和环境的可用性进行了实验评估。这些合作实验非常成功，因为研究生物学家发现它们非常适合工作环境，并积极采用它们进行现场研究计划。合作还证实，网格解决方案的计算性能得到了显着提高。高度的可用性和增强的计算性能的综合效果已经促进了生物实验方法的重大转变，从而提高了研究生产力。然而，由于基础设施本身的物理限制，在物理基础设施上实现的固定网格上实现的性能是有限的。云爆，即。在特定时间按需提供额外计算能力以科普研究计算的不可预测峰值的能力是云计算的一个有吸引力的方面，其具有突破在固定网格上可实现的当前性能约束的潜力。建议将云基础设施连接到现有的固定网格，以实现包括现有物理和虚拟化云资源的混合网格基础设施。然后建议重新设计工具和环境，以利用混合网格中的云爆发来实现更高的性能。最后，将评估混合物理/虚拟网格的性能，包括资源利用率的统计数据，以告知成本计算性能模型的开发。

Research scientists need cloud computing to flexibly support their computational requirements but also require user-friendly tools in order to engage. Demanding computational requirements for science research have been addressed through grid computing, which until recently has been based on a fixed physical wide-area infrastructure. Physical grid resources can become overloaded however - scientists may find this restricts their needs where the performance required at a particular moment in time cannot be addressed in real-time. User-friendly tools and environments are required to support grid computing in science communities. P-GRADE is a web-based portal, co-developed and by the investigators, for designing, submitting and monitoring workflows on the grid. It is highly graphical, and thus very appealing to a wide range of non-computing specialists.Familiarity is important to users. In order to develop the potential for cloud computing for the same community of science researchers, it is highly desirable, from their perspective, to retain continuity of computational support environment, as much as possible. The P-GRADE environment has been influenced by user requirements and aspirations over many years, and has already achieved a high degree of acceptance within the research science community, in view of its usability. In any case, to redesign a user interface from scratch is a very costly development activity.Interoperability between service grids and cluster grids was achieved by the investigators in the EU FP7-funded EDGeS project, so that users of each community may now access the resources of the other. Grid computing can also be extended onto cloud resources. The investigators have recently developed solutions for extending both service and cluster grids with virtual cloud resources.The usability of the tools and environment developed for the grid has been experimentally evaluated by the investigators in collaboration with biological research groups at the University of Westminster and Imperial College. These collaborative experiments have been very successful, in that the research biologists have found them very acceptable working environments, and have actively adopted them for their live research programmes. The collaborations have also confirmed that the computational performance of the grid solutions was significantly enhanced. The combined effect of a high degree of usability and enhanced computational performance has been to facilitate significant shifts in biological experimental methodology, leading to increased research productivity.Nevertheless, there are limits to the performance achievable on fixed grid implemented on physical infrastructure, due to the physical limitations of the infrastructure itself. Cloudbursting, ie. the ability to provide additional compute capacity that may be required on demand at specific times to cope with unpredictable peaks of research computing, is an attractive aspect of cloud computing which has the potential to break through the current performance constraints achievable on fixed grids. It is proposed to attach a cloud infrastructure to existing fixed grids to achieve a mixed grid infrastructure comprising both existing physical and virtualised cloud resources. It is then proposed to re-engineer the tools and environment to exploit cloudbursting in the mixed grid to achieve even greater performance. Finally, the performance of the mixed physical/virtual grid will be evaluated, to include statistics on resource utilisation, to inform the development of a costed performance model.