CAREER: Efficient and Reliable Data Transfer Services for Next Generation Research Networks

研究网络对于数据密集型、分布式和协作的科学项目至关重要，因为它们在研究机构和教育机构之间提供高速连接。然而，研究网络的用户不能有效地利用可用的资源，因为现有的传输应用在高速时受到可扩展性问题的影响。该项目为下一代研究网络设计和开发了一个可扩展的、健壮的数据传输框架，以提高其利用率。研究网络中增强的网络性能允许无缝执行下一代分布式科学应用程序，从而为快速取得突破性发现铺平道路。该项目还通过减少共享数据所需的时间来促进地理上不同机构的科学家之间的合作。除了研究贡献，该项目还将强有力的教育计划紧密地整合到其研究计划中。该计划包括为科学家提供培训，帮助他们在处理大规模数据时更好地利用先进的网络基础设施资源，为初中生和高中生开发游戏，教授网络概念，以及为代表不足的群体开设暑期学校，教授编程和网络。随着数据密集型分布式科学的趋势继续下去，开发能够扩展到下一代太比特每秒网络甚至更远的数据传输服务变得越来越重要。为了实现这一目标，本项目重点研究了四个方向：第一，创新了模块化文件传输体系结构，将I/O操作与网络传输分离，实现了动态和特定于组件的调优。其次，它实现了对延迟敏感的分布式工作流的服务质量支持，以满足其严格的性能要求。第三，它通过网络缓存/计算和概率错误检查机制为文件传输开发可扩展、安全和低开销的完整性验证。第四，它将开发的算法集成到常用的工作流管理系统中，以增加其被更广泛的科学界采用。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Research networks are crucial for data intensive, distributed, and collaborative science projects as they provide high speed connectivity between research and education institutions. However, users of research networks are unable to efficiently utilize available resources as existing transfer applications suffer from scalability issues at high speeds. This project designs and develops a scalable and robust data transfer framework for next-generation research networks to improve their utilization. Enhanced network performance in research networks allows seamless execution of next generation distributed science applications, thereby paving the way for breakthrough discoveries to be made swiftly. This project also promotes collaboration between scientists at geographically separated institutions by means of reducing the time it takes to share data. In addition to research contributions, this project has strong education plan tightly integrated into its research plan. The plan involves trainings for scientists to help them better utilize advanced cyberinfrastructure resources when dealing with large scale data, game development for middle and high school students to teach networking concepts, and summer schools for high school students for underrepresented groups to teach programming and networking. As trend towards data intensive distributed science continues, it is becoming increasingly important to develop data transfer services that can scale to next generation terabit per second networks and beyond. To achieve this goal, this project focuses four key research directions: First, it innovates a modular file transfer architecture to separate I/O operations from network transfers to enable dynamic and component specific tuning. Second, it implements Quality of Service support for delay sensitive distributed workflows to meet their stringent performance requirements. Third, it develops scalable, secure, and low overhead integrity verification for file transfers through in network caching/computing and probabilistic error checking mechanisms. Fourth, it integrates the developed algorithms to commonly used workflow management systems to increase its adoption by a broader science community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.