CC*IIE Integration: Collaborative Research: EPSON: Embracing Parallel Networks and Storage for Predictable End-to-End Data Movement

地理上分布的科学界需要越来越复杂的数据传输机制，以应对在异构网络上共享大型数据集的挑战。这些挑战包括优化具有不同配置和协议的网络，高效读写并行存储的I/O机制，以及不同数据传输工作负载的不同需求。为了应对这些挑战，爱普生项目正在开发、实施和评估应用编程接口和工具，以促进端到端并行数据传输。爱普生的研究人员专注于三个领域：（1）通过考虑共享网络和基础设施的并行网络特性的多样性以及专用电路和路径，实现并行网络数据移动，并有效平衡路径之间的流量，以实现更可预测的性能;（2）开发GridFTP数据存储接口，使并行文件系统之间的I/O可扩展，这对于校园基础设施处理大规模数据集至关重要;以及（3）设计将网络传输与存储I/O重叠的机制，并结合数据分级策略，匹配存储和网络能力之间的阻抗，以改善端到端数据传输。该项目涉及与应用科学家的密切合作，目的是提供先进的网络工具，以支持部署在芝加哥大学和北方伊利诺伊大学校园的应用程序的要求。

Geographically distributed scientific communities require increasingly sophisticated data transfer mechanisms that can handle the challenges of sharing large datasets over heterogeneous networks. These challenges include optimization of networks with different configurations and protocols, I/O mechanisms to efficiently read and write to parallel storage, and the varying demands of widely different data transfer workloads.To address these challenges, the EPSON project is developing, implementing, and evaluating application programming interfaces and tools that facilitate end-to-end parallel data transfers. EPSON researchers focus on three areas: (1) enabling parallel network data movement, by taking into account the diversity of parallel network characteristics of both shared networks and infrastructures with dedicated circuits and paths and effectively balancing the flows among paths for more predictable performance; (2) developing a GridFTP data storage interface, enabling scalable I/O to and from parallel filesystems -- critical for campus infrastructures to deal with large-scale datasets; and (3) devising mechanisms that overlap network transfers with storage I/O and incorporate data-staging heuristics, matching the impedance between storage and networking capabilities to improve end-to-end data transfers. The project involves close collaboration with application scientists, with the objective of providing advanced networking tools to support the requirements of the applications deployed at the University of Chicago and Northern Illinois University campuses.