DC: Medium: Intelligent Data Placement in Support of Scientific Workflows

变革性研究是通过对TB和PB范围内的大型数据集进行计算分析来进行的。这些分析通常是由科学工作流实现的，科学工作流在校园和国家网络基础设施资源上提供自动化和高效可靠的执行。工作流面临着许多与数据管理相关的问题，例如定位输入数据、找到与计算能力共存的必要存储，以及高效地暂存数据以使计算继续进行但存储资源不会被填满。这样的数据放置决策需要在单个工作流的环境中以及跨多个并发工作流进行。科学协作还需要执行数据放置操作，以传播和复制关键数据集。当多个科学协作共享网络基础设施并争夺有限的存储和计算资源时，就会出现额外的挑战。本项目将探讨这些情景下数据管理和计算管理之间的相互作用。该项目将包括设计支持大规模数据管理的算法和方法，以便有效地进行基于工作流程的计算，包括个别分析和工作流程组合，同时保留管理数据存储和访问的政策。将评估这些算法对在模拟和物理网络基础设施中运行的合成和真实世界工作流性能的影响。数据和计算管理的新方法可能会改变以千万亿级进行科学分析的方式。除了推进计算机科学，这项工作还将对一系列科学学科的数据和计算管理产生直接影响，这些学科管理大型数据集，并将它们用于在网络基础设施上运行的复杂分析。

Transformative research is conducted via computational analyses of large data sets in the terabyte and petabyte range. These analyses are often enabled by scientific workflows, which provide automation and efficient and reliable execution on campus and national cyberinfrastructure resources. Workflows face many issues related to data management such as locating input data, finding necessary storage co-located with computing capabilities, and efficiently staging data so that the computation progresses but storage resources do not fill up. Such data placement decisions need to be made within the context of individual workflows and across multiple concurrent workflows. Scientific collaborations also need to perform data placement operations to disseminate and replicate key data sets. Additional challenges arise when multiple scientific collaborations share cyberinfrastructure and compete for limited storage and compute resources. This project will explore the interplay between data management and computation management for these scenarios. The project will include the design of algorithms and methodologies that support large-scale data management for efficient workflow-based computations composed of individual analyses and workflow ensembles while preserving policies governing data storage and access. The algorithms will be evaluated regarding their impact on performance of synthetic and real-world workflows running in simulated and physical cyberinfrastructures. New approaches to data and computation management can potentially transform how scientific analyses are conducted at the petascale. Besides advancing computer science, this work will have direct impact on data and computation management for a range of scientific disciplines that manage large data sets and use them in complex analyses running on cyberinfrastructure.