Collaborative Research SI2 SSE: Pipeline Framework for Ensemble Runs on Clouds

协作研究 SI2 SSE：云上运行的 Ensemble 管道框架

• 批准号: 1148359
• 负责人: Beth Plale
• 金额: $ 29.29万
• 依托单位: Indiana University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2012
• 资助国家: 美国
• 起止时间: 2012-05-01 至 2014-04-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1148359&HistoricalAwards=false
• 关键词: Collaborative; Research; SI2; SSE; Pipeline

Cloud computing is an attractive computational resource for e-Science because of the ease with which cores can be accessed on demand, and because the virtual machine implementation that underlies cloud computing reduces the cost of porting a numeric or analysis code to a new platform. It is difficult to use cloud computing resources for large-scale, high throughput ensemble jobs however. Additionally, the computationally oriented researcher is increasingly encouraged to make data sets available to the broader community. For the latter to be achieved, using capture tools during experimentation to harvest metadata and provenance reduces the manual burden of marking up results. Better automatic capture of metadata and provenance is the only means by which sharing of scientific data can scale to meet the burgeoning explosion of data.This project develops a pipeline framework for running ensemble simulations on the cloud; the framework has two key components: ensemble deployment and metadata harvest. Regarding the former, on commercial cloud platforms typically a much smaller number of jobs than desired can be started at any one time. An ensemble run will need to be pipelined to a cloud resource, that is, executed in well-controlled batches over a period of time. We will use platform features of Azure, and employ machine learning techniques to continuously refine the pipeline submission strategy and workflow strategies for ensemble parameter specification, pipelined deployment, and metadata capture. Regarding the latter key component, we expect to reduce the burden of sharing scientific datasets resulting from the use of cloud resources through automatic metadata and provenance capture and representation that aligns the metadata with emerging best practices in data sharing and discovery. Ensemble simulations result in complex data sets, whose reuse could be increased by expressive, granule and collection level metadata, including the lineage of the resulting products, to contribute towards trust.In this project we focus on a compelling and timely application from climate research: One of the more immediate and dangerous impacts of climate change could be a change in the strength of storms that form over the oceans. In addition, as sea level rises due to global warming and melting of the polar ice caps, coastal communities will become increasingly vulnerable to storm surge. There have already been indications that even modest changes in ocean surface temperature can have a disproportionate effect on hurricane strength and the damage inflicted by these storms. In an effort to understand these impacts, modelers turn to predictions generated by hydrodynamic coastal ocean models such as the Sea, Lake and Overland Surges from Hurricanes (SLOSH) model. The proposed research advances the knowledge and understanding of probabilistic storm surge products by enhancements to the SLOSH model itself and through mechanisms that take advantage of commercial cloud resources. This knowledge is expected to have application in research, the classroom, and in operational settings.The broader significance of the project is several-fold. Cloud computing is an important economic driver but it remains difficult for use in computationally driven scientific research. This project lowers the barriers to conducting e-Science research that utilizes cloud resources, specifically Azure. It will contribute tools to help researchers share, preserve, and publicize the scientific data sets that result from their research. Because we focus on and improve an application that predicts storm surge in response to sea level changes and severe storms, our work contributes to societal responses and adaptations to climate change, including planning and building the sustainable, hazard-resilient coastal communities of the future.

云计算对于e-Science来说是一种有吸引力的计算资源，因为可以根据需要轻松访问核心，并且因为作为云计算基础的虚拟机实现降低了将数值或分析代码移植到新平台的成本。然而，很难将云计算资源用于大规模、高吞吐量的集成作业。此外，越来越多地鼓励面向计算的研究人员将数据集提供给更广泛的社区。为了实现后者，在实验过程中使用捕获工具来收集元数据和出处减少了标记结果的手动负担。更好地自动捕获元数据和来源是科学数据共享可以扩展以满足数据爆炸的唯一手段。该项目开发了一个用于在云上运行集成模拟的管道框架;该框架有两个关键组件：集成部署和元数据收获。关于前者，在商业云平台上，通常一次可以启动的作业数量比期望的要少得多。集成运行需要通过管道传输到云资源，即在一段时间内以良好控制的批处理方式执行。我们将使用Azure的平台功能，并采用机器学习技术来不断完善管道提交策略和工作流策略，以实现集成参数规范、管道部署和元数据捕获。关于后一个关键组成部分，我们希望通过自动元数据和来源捕获和表示，使元数据与数据共享和发现方面的新兴最佳做法保持一致，从而减轻因使用云资源而共享科学数据集的负担。Ensemble模拟产生复杂的数据集，其重用可以通过表达，颗粒和收集级别的元数据来增加，包括产生的产品的血统，以促进信任。在这个项目中，我们专注于气候研究的一个引人注目的和及时的应用：气候变化的一个更直接和危险的影响可能是在海洋上形成的风暴强度的变化。此外，随着全球变暖和极地冰盖融化导致海平面上升，沿海社区将越来越容易受到风暴潮的影响。已经有迹象表明，即使海洋表面温度发生轻微变化，也会对飓风强度和这些风暴造成的破坏产生不成比例的影响。为了理解这些影响，建模者转向流体动力学沿海海洋模型，如飓风引起的海洋、湖泊和陆上涌浪（SLOSH）模型。拟议的研究通过增强SLOSH模型本身并通过利用商业云资源的机制来提高对概率风暴潮产品的认识和理解。这些知识有望在研究、课堂和操作环境中得到应用。该项目的更广泛意义是多重的。云计算是一个重要的经济驱动力，但它仍然难以用于计算驱动的科学研究。该项目降低了利用云资源（特别是Azure）进行电子科学研究的障碍。它将提供工具，帮助研究人员分享、保存和公布他们的研究所产生的科学数据集。因为我们专注于并改进了一个应用程序，预测风暴潮以应对海平面变化和严重风暴，我们的工作有助于社会应对和适应气候变化，包括规划和建设未来可持续的，具有抗灾能力的沿海社区。