EAGER: Fundamental Issues in International Data Placement for Data-Intensive Applications, a Laboratory Approach

从天文学到医学，从经济学到植物学，越来越多的科学学科的研究人员希望跨越机构和国家边界移动大量数据。将数据放置在不同的位置可以扩展数据处理能力，促进协作分析，并加速将获得的见解转化为科学发现。然而，我们对相互关联的复杂因素如何影响这些数据移动的成本、性能和资源消耗知之甚少。这些因素包括数据集中文件的数量和大小分布、源和目标文件/存储系统的性能特征、站点之间网络承载能力的时间/周波动、IPv6路径的可用性。 该项目将探索模型，捕捉的特点和相互作用的方面，参与放置数据在地方，国家或国际网络分开的位置。将开展探索性工作，为国际数据布局实验室（iDPL）建立组织和功能基础，以支持大规模的端到端数据布局实验。iDPL将与国际合作伙伴一起建造，旨在成为一个可扩展的设施。 UCSD和UW-Madison团队汇集了软件工具（HTCondor，Open Science Grid，Rocks Clustering），高性能校园网络（Quartzite和Prism@UCSD），存储系统（Data Oasis）和参与科学社区的丰富经验。 iDPL的核心软件架构是将数据放置实验视为工作流，然后使用HTCondor？的DAGMan和Metronome连续测试框架来编写、实例化和管理实验。 将提供合成数据集，以便利诊断。 只要有可能，国际网络提供商将参与将网络的基础设施视图与用户体验相关联。广泛和长期的影响是为持久的数据和信息共享工具奠定基础，该工具可以轻松扩展到最初的国际机构集团之外

Researchers in an expanding number of scientific disciplines, ranging from astronomy to medicine and from economics, to botany, would like to move large volumes of data across institutional and national boundaries. Placing data at different locations expands data processing capacity, facilitates collaborative analysis and accelerates translation of gained insight into scientific discoveries. Yet, we have little understanding of how a complex ensemble of interrelated factors affect the cost, performance, and resource consumption of these data movements. Factors include the number and size distributions of files in a data set, performance characteristics of the source and destination file/storage systems, time-of-day/week fluctuations of the network carrying capacity between sites, availability of IPv6 paths. This project will explore models that capture both the characteristics and interplay of facets that are involved in placing data at locations separated by local, national, or international networks. Exploratory work will take place to establish organizational and functional foundations for an international data placement laboratory (iDPL) to support at-scale end-to-end data placement experiments.The iDPL will be constructed with international partners and is intended to be an extensible facility. The UCSD and UW-Madison team bring together extensive experience in software tools (HTCondor, Open Science Grid, Rocks Clustering), high-performance campus-area networks (Quartzite and Prism@UCSD), storage systems (Data Oasis), and engaged science communities. The iDPL core software architecture is to treat data placement experiments as workflows and then use HTCondor?s DAGMan and Metronome continuous testing framework to author, instantiate and manage experiments. Synthetic data sets will be offered to facilitate diagnostics. Whenever possible, international networking providers will be engaged to correlate infrastructure view of the network with user experiences.The broad and long term impact is to lay the groundwork for a persistent, data and information sharing vehicle that can be easily expanded beyond the initial international group of institutions