DC: Medium: Intelligent Data Placement in Support of Scientific Workflows

变革性研究是通过对 TB 和 PB 范围内的大型数据集进行计算分析来进行的。这些分析通常通过科学的工作流程来实现，这些工作流程在校园和国家网络基础设施资源上提供自动化以及高效可靠的执行。工作流面临着与数据管理相关的许多问题，例如定位输入数据、找到与计算功能位于同一位置的必要存储以及有效地暂存数据，以便计算继续进行但存储资源不会填满。此类数据放置决策需要在各个工作流程的上下文中以及跨多个并发工作流程做出。科学合作还需要执行数据放置操作来传播和复制关键数据集。当多个科学合作共享网络基础设施并争夺有限的存储和计算资源时，就会出现额外的挑战。该项目将探讨这些场景的数据管理和计算管理之间的相互作用。该项目将包括算法和方法的设计，支持大规模数据管理，以实现由单独分析和工作流集合组成的高效基于工作流的计算，同时保留管理数据存储和访问的策略。将评估这些算法对在模拟和物理网络基础设施中运行的合成和现实工作流程的性能的影响。数据和计算管理的新方法可能会改变千万亿级科学分析的方式。除了推进计算机科学之外，这项工作还将对管理大型数据集并将其用于在网络基础设施上运行的复杂分析的一系列科学学科的数据和计算管理产生直接影响。

Transformative research is conducted via computational analyses of large data sets in the terabyte and petabyte range. These analyses are often enabled by scientific workflows, which provide automation and efficient and reliable execution on campus and national cyberinfrastructure resources. Workflows face many issues related to data management such as locating input data, finding necessary storage co-located with computing capabilities, and efficiently staging data so that the computation progresses but storage resources do not fill up. Such data placement decisions need to be made within the context of individual workflows and across multiple concurrent workflows. Scientific collaborations also need to perform data placement operations to disseminate and replicate key data sets. Additional challenges arise when multiple scientific collaborations share cyberinfrastructure and compete for limited storage and compute resources. This project will explore the interplay between data management and computation management for these scenarios. The project will include the design of algorithms and methodologies that support large-scale data management for efficient workflow-based computations composed of individual analyses and workflow ensembles while preserving policies governing data storage and access. The algorithms will be evaluated regarding their impact on performance of synthetic and real-world workflows running in simulated and physical cyberinfrastructures. New approaches to data and computation management can potentially transform how scientific analyses are conducted at the petascale. Besides advancing computer science, this work will have direct impact on data and computation management for a range of scientific disciplines that manage large data sets and use them in complex analyses running on cyberinfrastructure.