CSR: Small: Collaborative Research: An Integrated Approach to Performance Modeling and Optimization of Big-data Scientific Workflows

下一代电子科学正在产生大量的数据，通常被称为大数据，目前的数量级为太字节，在可预见的未来，数量级为拍字节甚至是艾字节。这些科学应用程序通常具有数据和网络密集型工作流，这些工作流由具有复杂的模块间依赖关系的计算模块组成。应用程序用户经常需要在分布式环境中以一种特别的方式手动配置他们的计算工作流，这极大地限制了科学家的生产力并限制了资源的利用。大数据科学工作流的端到端性能既取决于决定模块分配的映射方案，也取决于决定资源分配的调度策略。基于工作流的研究过程的这两个方面传统上被视为两个独立的主题，它们之间的相互作用尚未得到充分探讨。随着科学工作流程和网络环境的规模和复杂性迅速增加，性能优化的每个方面都取得了有限的成功。本研究对资源共享环境下工作流执行动态进行深入研究，探讨统一应用支持平台下工作流映射与节点调度之间的相互作用。其思想是在严格的算法设计、理论动态分析和实际网络实施、部署和评估的基础上，构建一个无缝集成三个相互关联组件的三层工作流优化架构。该项目的成功完成将为大数据科学工作流的系统动力学分析和控制提供坚实的数学基础，生成一套协作映射和调度优化解决方案，以促进科学协作，并为当前网格和云计算中间件中广泛采用的现有工作流引擎增加额外的智能水平。由此产生的工作流优化解决方案将有利于广泛的基于工作流的科学应用

Next-generation e-science is producing colossal amounts of data, commonly known as Big Data, on the order of terabyte at present and petabyte or even exabyte in the predictable future. These scientific applications typically feature data- and network-intensive workflows comprised of computing modules with intricate inter-module dependencies. Application users oftentimes need to manually configure their computing workflows in distributed environments in an ad-hoc manner, which significantly limits the productivity of scientists and constrains the utilization of resources.The end-to-end performance of big data scientific workflows depends on both the mapping scheme that determines module assignment and the scheduling policy that determines resource allocation. These two aspects of a workflow-based research process are traditionally treated as two separate topics, and the interactions between them have not been fully explored. As the scale and complexity of scientific workflows and network environments rapidly increase, each individual aspect of performance optimization has limited success. This research is an in-depth investigation into workflow execution dynamics in resource sharing environments to explore the interactions between workflow mapping and node scheduling on a unified application-support platform. The idea is to build a three-layer workflow optimization architecture that seamlessly integrates three interrelated components based on rigorous algorithmic design, theoretical dynamics analysis, and real network implementation, deployment, and evaluation. The successful completion of this project will provide a solid mathematical foundation for the analysis and control of system dynamics of big data scientific workflows, produce a suite of cooperative mapping and scheduling optimization solutions to facilitate scientific collaborations, and add an additional level of intelligence to existing workflow engines widely adopted in the current grid and cloud computing middleware. The resulting workflow optimization solutions will benefit a broad spectrum of workflow-based scientific applications