CSR: Small: Collaborative Research: An Integrated Approach to Performance Modeling and Optimization of Big-data Scientific Workflows

CSR：小型：协作研究：大数据科学工作流程性能建模和优化的综合方法

• 批准号: 1525537
• 负责人: Yi Gu
• 金额: $ 12.5万
• 依托单位: Middle Tennessee State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2015
• 资助国家: 美国
• 起止时间: 2015-10-01 至 2018-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1525537&HistoricalAwards=false
• 关键词: CSR; Small; Collaborative; Research; Integrated

Next-generation e-science is producing colossal amounts of data, commonly known as Big Data, on the order of terabyte at present and petabyte or even exabyte in the predictable future. These scientific applications typically feature data- and network-intensive workflows comprised of computing modules with intricate inter-module dependencies. Application users oftentimes need to manually configure their computing workflows in distributed environments in an ad-hoc manner, which significantly limits the productivity of scientists and constrains the utilization of resources.The end-to-end performance of big data scientific workflows depends on both the mapping scheme that determines module assignment and the scheduling policy that determines resource allocation. These two aspects of a workflow-based research process are traditionally treated as two separate topics, and the interactions between them have not been fully explored. As the scale and complexity of scientific workflows and network environments rapidly increase, each individual aspect of performance optimization has limited success. This research is an in-depth investigation into workflow execution dynamics in resource sharing environments to explore the interactions between workflow mapping and node scheduling on a unified application-support platform. The idea is to build a three-layer workflow optimization architecture that seamlessly integrates three interrelated components based on rigorous algorithmic design, theoretical dynamics analysis, and real network implementation, deployment, and evaluation. The successful completion of this project will provide a solid mathematical foundation for the analysis and control of system dynamics of big data scientific workflows, produce a suite of cooperative mapping and scheduling optimization solutions to facilitate scientific collaborations, and add an additional level of intelligence to existing workflow engines widely adopted in the current grid and cloud computing middleware. The resulting workflow optimization solutions will benefit a broad spectrum of workflow-based scientific applications

在可预测的未来，下一代电子科学目前正在生产大量的数据（通常称为大数据），而PBABYTE甚至是Apabyte。这些科学应用程序通常具有数据和网络密集型工作流程，包括具有复杂模块间依赖性的计算模块。应用程序用户通常需要以临时的方式手动在分布式环境中手动配置其计算工作流程，这显着限制了科学家的生产力并限制了资源的利用。传统上将基于工作流的研究过程的这两个方面视为两个独立的主题，并且尚未完全探索它们之间的相互作用。随着科学工作流和网络环境的规模和复杂性迅速增加，性能优化的每个方面都取得了有限的成功。这项研究是对资源共享环境中工作流执行动态的深入研究，以探索统一的应用程序支持平台上工作流映射与节点调度之间的交互。这个想法是建立一个三层工作流优化架构，该体系结构无缝地集成了基于严格的算法设计，理论动态分析以及实际网络实现，部署和评估的三个相互关联的组件。该项目的成功完成将为分析和控制大数据科学工作流程的系统动态，生产一套合作映射和调度优化解决方案，以促进科学合作，并为当前的网格和云计算中的现有工作流程添加额外的智能水平。 由此产生的工作流优化解决方案将受益于广泛的基于工作流程的科学应用