CIF21 DIBBs: PD: OneDataShare: A Universal Data Sharing Building Block for Data-Intensive Applications

CIF21 DIBB：PD：OneDataShare：数据密集型应用程序的通用数据共享构建块

• 批准号: 1724898
• 负责人: Tevfik Kosar
• 金额: $ 49.78万
• 依托单位: SUNY at Buffalo
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2017
• 资助国家: 美国
• 起止时间: 2017-09-01 至 2022-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1724898&HistoricalAwards=false
• 关键词: CIF21; DIBBs; PD; OneDataShare; Universal

Applications in scientific, industrial, and personal spaces now generate more data than ever before. As data become more abundant and data resources become more heterogeneous, the accessing, sharing and disseminating of data sets becomes a bigger challenge. Existing technologies for transferring and sharing data suffer from serious shortcomings, including low transfer performance, inflexibility, restricted protocol support, and poor scalability. This project develops a universal data sharing building block for data-intensive applications, dubbed OneDataShare, with three major goals: (1) optimization of end-to-end data transfers and reduction of the time to delivery of the data; (2) interoperation across heterogeneous and incompatible data resources; and (3) predicting the data delivery time and decreasing the uncertainty in real-time decision-making processes.OneDataShare deliverables include: (1) design and implementation of novel algorithms for application-layer optimization of the data transfer protocol parameters to achieve optimal end-to-end data transfer throughput; (2) development of a universal interface specification for heterogeneous data storage endpoints and a framework for on-the-fly data transfer protocol translation; (3) instrumentation of end-to-end data transfer time prediction capability, and feeding it into real-time scheduling and decision-making processes for advanced provisioning, high-level planning, and co-scheduling of resources; (4) deployment of these capabilities as stand-alone OneDataShare cloud-hosted services to end users; and (5) integration of these capabilities with widely used data scheduling and workflow management tools, and validation in specific applications. OneDataShare services and tools are developed at the application level, and they do not require any changes to the existing infrastructure, nor to the low-level networking stack, although they increase the end-to-end performance of the data movement tasks substantially. These efficient and high-performance data transfer techniques will help the scientific community, industry, and end-users to save significant time and effort in transferring and sharing data.

科学、工业和个人空间中的应用现在产生的数据比以往任何时候都多。随着数据的日益丰富和数据资源的日益异构，数据集的访问、共享和传播成为一个更大的挑战。 现有的数据传输和共享技术存在严重的缺陷，包括传输性能低、可扩展性差、协议支持受限以及可扩展性差。 该项目为数据密集型应用程序开发了一个通用的数据共享构建块，称为OneDataShare，有三个主要目标：（1）优化端到端的数据传输和减少数据交付的时间;（2）跨异构和不兼容的数据资源的互操作;以及（3）预测数据交付时间，减少实时决策过程中的不确定性。OneDataShare可交付成果包括：（1）设计和实现用于数据传输协议参数的应用层优化的新颖算法，以实现最佳的端到端数据传输吞吐量;（2）开发用于异构数据存储端点的通用接口规范和用于实时数据传输协议翻译的框架;（3）端到端数据传输时间预测能力的仪表化，并将其馈送到实时调度和决策制定过程中，以用于资源的高级供应、高级规划和协同调度;（4）将这些能力作为独立的OneDataShare云托管服务部署到最终用户;以及（5）将这些能力与广泛使用的数据调度和工作流管理工具集成，并在特定应用中进行验证。 OneDataShare服务和工具是在应用程序级别开发的，它们不需要对现有基础设施进行任何更改，也不需要对低级网络堆栈进行任何更改，尽管它们大大提高了数据移动任务的端到端性能。 这些高效和高性能的数据传输技术将帮助科学界、工业界和最终用户在传输和共享数据方面节省大量时间和精力。

项目成果

GreenDataFlow: Minimizing the Energy Footprint of Global Data Movement

• DOI: 10.1109/bigdata.2018.8622570
• 发表时间: 2018-10
• 期刊: 2018 IEEE International Conference on Big Data (Big Data)
• 作者: M. S. Q. Z. Nine;Luigi Di Tacchio;A. Imran;T. Kosar;Muhammed Fatih Bulut;Jinho Hwang

Energy-saving Cross-layer Optimization of Big Data Transfer Based on Historical Log Analysis

基于历史日志分析的大数据传输节能跨层优化

• DOI: 10.1109/icc42927.2021.9500693
• 发表时间: 2021
• 期刊: ICC 2021 - IEEE International Conference on Communications
• 作者: Rodolph, Lavone;Zulkar Nine, MD S;Di Tacchio, Luigi;Kosar, Tevfik
• 通讯作者: Kosar, Tevfik

High-Speed Transfer Optimization Based on Historical Analysis and Real-Time Tuning

基于历史分析和实时调优的高速传输优化

• DOI: 10.1109/tpds.2018.2790948
• 发表时间: 2018
• 期刊: IEEE Transactions on Parallel and Distributed Systems
• 影响因子: 5.3
• 作者: Arslan, Engin;Kosar, Tevfik
• 通讯作者: Kosar, Tevfik

Big data transfer optimization based on offline knowledge discovery and adaptive sampling

• DOI: 10.1109/bigdata.2017.8257959
• 发表时间: 2017-12
• 期刊: 2017 IEEE International Conference on Big Data (Big Data)
• 作者: M. S. Q. Z. Nine;Kemal Guner;Ziyun Huang;Xiangyu Wang;Jinhui Xu;T. Kosar

Cross-Layer Optimization of Big Data Transfer Throughput and Energy Consumption

• DOI: 10.1109/cloud.2019.00017
• 发表时间: 2019-07
• 期刊: 2019 IEEE 12th International Conference on Cloud Computing (CLOUD)
• 作者: Luigi Di Tacchio;M. S. Q. Z. Nine;T. Kosar;Muhammed Fatih Bulut;Jinho Hwang