Scalable Data Coupling Abstraction for Data-Intensive Simulation Workflows

大规模耦合仿真工作流的可扩展数据管理抽象耦合的科学仿真工作流集成了多个物理和尺度，并在高端资源上以非常大的规模运行，具有实现前所未有的精度水平并提供对复杂现象的戏剧性见解的潜力。然而，这些仿真工作流的耦合组件需要在运行时交互和交换大量数据，并且数据在从源流向目的地时通常必须进行转换。随着这些数据的数量和产生率的增长，与提取这些数据并将其传输用于耦合、转换和分析相关的成本(延迟和能量)已经成为主要的管理费用，并且决定了可以达到的性能和生产率的水平。本项目的目标是解决这些挑战，开发概念性解决方案以及能够实现大规模数据密集型模拟的软件框架。我们的方法是基于这样一个前提，即考虑到大量的数据量和相关的成本，数据将不得不在很大程度上在线、现场、？然后呢？在运输途中？虽然它是使用计算平台内的资源来分阶段的，并且编程和运行时系统必须提供促进这种数据处理的抽象和机制。我们的努力围绕着三个关键的研究主题：(1)现场/在途数据管理的编程抽象；(2)可扩展数据准备基板的设计和实现；以及(3)以数据为中心的映射和调度。数据和计算密集型模拟对广泛的科学和工程领域越来越重要，因此，这项研究具有推动这些领域研究和创新的潜力。开发的框架和基准还为计算机科学家提供了实验和探索以数据为中心的研究的基础。人力资源的开发，包括对学生、研究人员和软件专业人员的培训，以及与少数群体和任职人数不足的群体的接触，是这项努力的所有方面不可或缺的一部分。

A Scalable Data Management Abstraction for Large-scale Coupled Simulation WorkflowsCoupled scientific simulation workflows, integrating multiple physics and scales and running at very large scales on high-end resources, have the potential for achieving unprecedented levels of accuracy and providing dramatic insights into complex phenomena. However, the coupled component of these simulation workflows need to interact and exchange significant amounts of data at runtime, and the data often has to be transformed as it flows from source to destination. As the volumes and generation rates of this data grow, the costs (latencies and energy) associated with extracting this data and transporting it for coupling, transformation and analysis have become the dominating overheads and are dictating the level of performance and productivity that can be achieved.The goal of this project is to address these challenges and to develop conceptual solutions as well as a software framework that can enable the large-scale data-intensive simulations. Our approach is based on the premise that given the large data volumes and associated costs, data will have to be largely processed online, ?in-situ? and ?in-transit? while it is staged using resources within the computational platform, and the programming and runtime system must provide abstractions and mechanisms that facilitate such data processing. Our effort is organized around three key research thrusts: (1) Programming abstractions for in-situ/in-transit data management; (2) Design and implementation of a scalable data staging substrate; and (3) Data-centric mapping and scheduling.Data and compute intensive simulations are becoming increasingly critical to a wide range of science and engineering domains, and as a result, this research has the potential to drive research and innovations in these domains. The developed framework and benchmarks also provide computer scientists with a substrate to experiment with and explore data-centric research. The development of human resources, including the training of students, researchers and software professions, as well as outreach to minorities and underrepresented group, is integral to all aspects of this effort.