CSR: Small: Collaborative Research: An Integrated Approach to Performance Modeling and Optimization of Big-data Scientific Workflows

下一代电子科学正在产生大量的数据，通常称为大数据，目前为TB级，在可预测的未来为PB级甚至EB级。这些科学应用程序通常具有数据和网络密集型工作流，这些工作流由具有复杂模块间依赖关系的计算模块组成。在分布式环境中，应用程序用户往往需要以自组织的方式手动配置计算工作流，这极大地限制了科学家的工作效率和资源利用率。大数据科学工作流的端到端性能取决于决定模块分配的映射方案和决定资源分配的调度策略。基于工作流的研究过程的这两个方面传统上被视为两个独立的主题，它们之间的相互作用尚未得到充分探讨。随着科学工作流和网络环境的规模和复杂性迅速增加，性能优化的每个方面都取得了有限的成功。本研究深入探讨资源共享环境下的工作流执行动力学，探讨统一应用支撑平台下工作流映射与节点调度之间的相互作用。其思想是建立一个三层的工作流优化架构，无缝集成三个相互关联的组件的基础上严格的算法设计，理论动力学分析，和真实的网络的实施，部署和评估。该项目的成功完成将为大数据科学工作流的系统动力学分析和控制提供坚实的数学基础，产生一套协作映射和调度优化解决方案，以促进科学合作，并为当前网格和云计算中间件中广泛采用的现有工作流引擎增加额外的智能水平。 由此产生的工作流优化解决方案将有利于广泛的基于工作流的科学应用

Next-generation e-science is producing colossal amounts of data, commonly known as Big Data, on the order of terabyte at present and petabyte or even exabyte in the predictable future. These scientific applications typically feature data- and network-intensive workflows comprised of computing modules with intricate inter-module dependencies. Application users oftentimes need to manually configure their computing workflows in distributed environments in an ad-hoc manner, which significantly limits the productivity of scientists and constrains the utilization of resources.The end-to-end performance of big data scientific workflows depends on both the mapping scheme that determines module assignment and the scheduling policy that determines resource allocation. These two aspects of a workflow-based research process are traditionally treated as two separate topics, and the interactions between them have not been fully explored. As the scale and complexity of scientific workflows and network environments rapidly increase, each individual aspect of performance optimization has limited success. This research is an in-depth investigation into workflow execution dynamics in resource sharing environments to explore the interactions between workflow mapping and node scheduling on a unified application-support platform. The idea is to build a three-layer workflow optimization architecture that seamlessly integrates three interrelated components based on rigorous algorithmic design, theoretical dynamics analysis, and real network implementation, deployment, and evaluation. The successful completion of this project will provide a solid mathematical foundation for the analysis and control of system dynamics of big data scientific workflows, produce a suite of cooperative mapping and scheduling optimization solutions to facilitate scientific collaborations, and add an additional level of intelligence to existing workflow engines widely adopted in the current grid and cloud computing middleware. The resulting workflow optimization solutions will benefit a broad spectrum of workflow-based scientific applications