CC*IIE Integration: Collaborative Research: EPSON: Embracing Parallel Networks and Storage for Predictable End-to-End Data Movement

CC*IIE 集成：协作研究：EPSON：采用并行网络和存储实现可预测的端到端数据移动

• 批准号: 1440761
• 负责人: Rajkumar Kettimuthu
• 金额: $ 36万
• 依托单位: University of Chicago
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2014
• 资助国家: 美国
• 起止时间: 2014-10-01 至 2017-09-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1440761&HistoricalAwards=false
• 关键词: CC; IIE; Integration; Collaborative; Research

Geographically distributed scientific communities require increasingly sophisticated data transfer mechanisms that can handle the challenges of sharing large datasets over heterogeneous networks. These challenges include optimization of networks with different configurations and protocols, I/O mechanisms to efficiently read and write to parallel storage, and the varying demands of widely different data transfer workloads.To address these challenges, the EPSON project is developing, implementing, and evaluating application programming interfaces and tools that facilitate end-to-end parallel data transfers. EPSON researchers focus on three areas: (1) enabling parallel network data movement, by taking into account the diversity of parallel network characteristics of both shared networks and infrastructures with dedicated circuits and paths and effectively balancing the flows among paths for more predictable performance; (2) developing a GridFTP data storage interface, enabling scalable I/O to and from parallel filesystems -- critical for campus infrastructures to deal with large-scale datasets; and (3) devising mechanisms that overlap network transfers with storage I/O and incorporate data-staging heuristics, matching the impedance between storage and networking capabilities to improve end-to-end data transfers. The project involves close collaboration with application scientists, with the objective of providing advanced networking tools to support the requirements of the applications deployed at the University of Chicago and Northern Illinois University campuses.

地理分布的科学社区需要越来越复杂的数据传输机制，以应对在异构网络上共享大型数据集的挑战。这些挑战包括使用不同配置和协议优化网络，有效读写并行存储的I/O机制，以及不同数据传输工作负载的不同需求。为了应对这些挑战，EPSON项目正在开发、实现和评估促进端到端并行数据传输的应用程序编程接口和工具。EPSON的研究人员关注三个方面：(1)通过考虑共享网络和专用电路和路径的基础设施的并行网络特性的多样性，并有效地平衡路径之间的流量，实现并行网络数据移动，从而获得更可预测的性能；(2)开发GridFTP数据存储接口，实现并行文件系统的可扩展I/O，这对于校园基础设施处理大规模数据集至关重要；(3)设计将网络传输与存储I/O重叠的机制，并结合数据分段启发式，匹配存储和网络功能之间的阻抗，以改善端到端数据传输。该项目涉及与应用科学家的密切合作，目标是提供先进的网络工具，以支持在芝加哥大学和北伊利诺伊大学校园部署的应用程序的需求。