CIF21 DIBBs: PD: OneDataShare: A Universal Data Sharing Building Block for Data-Intensive Applications

科学、工业和个人空间中的应用现在产生的数据比以往任何时候都多。随着数据的日益丰富和数据资源的日益异构，数据集的访问、共享和传播成为一个更大的挑战。 现有的数据传输和共享技术存在严重缺陷，包括传输性能低、不灵活、协议支持有限以及可扩展性差。 该项目为数据密集型应用程序开发了一个通用的数据共享构建块，称为OneDataShare，有三个主要目标：（1）优化端到端的数据传输和减少数据交付的时间;（2）跨异构和不兼容的数据资源的互操作;以及（3）预测数据交付时间，减少实时决策过程中的不确定性。OneDataShare可交付成果包括：（1）设计和实现用于数据传输协议参数的应用层优化的新颖算法，以实现最佳的端到端数据传输吞吐量;（2）开发用于异构数据存储端点的通用接口规范和用于实时数据传输协议翻译的框架;（3）端到端数据传输时间预测能力的仪表化，并将其馈送到实时调度和决策制定过程中，以用于资源的高级供应、高级规划和协同调度;（4）将这些能力作为独立的OneDataShare云托管服务部署到最终用户;以及（5）将这些能力与广泛使用的数据调度和工作流管理工具集成，并在特定应用中进行验证。 OneDataShare服务和工具是在应用程序级别开发的，它们不需要对现有基础设施进行任何更改，也不需要对低级网络堆栈进行任何更改，尽管它们大大提高了数据移动任务的端到端性能。 这些高效和高性能的数据传输技术将帮助科学界、工业界和最终用户在传输和共享数据方面节省大量时间和精力。

Applications in scientific, industrial, and personal spaces now generate more data than ever before. As data become more abundant and data resources become more heterogeneous, the accessing, sharing and disseminating of data sets becomes a bigger challenge. Existing technologies for transferring and sharing data suffer from serious shortcomings, including low transfer performance, inflexibility, restricted protocol support, and poor scalability. This project develops a universal data sharing building block for data-intensive applications, dubbed OneDataShare, with three major goals: (1) optimization of end-to-end data transfers and reduction of the time to delivery of the data; (2) interoperation across heterogeneous and incompatible data resources; and (3) predicting the data delivery time and decreasing the uncertainty in real-time decision-making processes.OneDataShare deliverables include: (1) design and implementation of novel algorithms for application-layer optimization of the data transfer protocol parameters to achieve optimal end-to-end data transfer throughput; (2) development of a universal interface specification for heterogeneous data storage endpoints and a framework for on-the-fly data transfer protocol translation; (3) instrumentation of end-to-end data transfer time prediction capability, and feeding it into real-time scheduling and decision-making processes for advanced provisioning, high-level planning, and co-scheduling of resources; (4) deployment of these capabilities as stand-alone OneDataShare cloud-hosted services to end users; and (5) integration of these capabilities with widely used data scheduling and workflow management tools, and validation in specific applications. OneDataShare services and tools are developed at the application level, and they do not require any changes to the existing infrastructure, nor to the low-level networking stack, although they increase the end-to-end performance of the data movement tasks substantially. These efficient and high-performance data transfer techniques will help the scientific community, industry, and end-users to save significant time and effort in transferring and sharing data.