Collaborative Research: CC-NIE Integration: A Data Movement Solution for Next-Generation Campus Cyberinfrastructure

研究进展往往严重依赖于远距离传输数据的能力：例如，从远程超级计算机、望远镜或基因组测序中心到研究人员的实验室或大学数据中心。然而，尽管广域网和校园网取得了重大进展，但由于防火墙和工具不足等障碍，数据移动仍然非常困难和缓慢。为了克服这个问题，该项目开发了一种易于使用的高性能数据传输和同步解决方案，可以在研究人员需要数据的任何地方运行：桌面和研究实验室、校园服务器和高性能计算设施、超级计算中心、科学仪器以及公共和私有云提供商。这种新的数据移动解决方案建立在广泛使用的GridFTP协议的Globus实现和较新的Globus Transfer托管数据移动服务，以及印第安纳大学可扩展会话协议（XSP）的实现之上，该协议通过OpenFlow等协议提供端到端电路的协商。它集成了许多新功能，包括端到端电路的协商；位于网络地址转换（NAT）设备或防火墙后面的节点对之间的通信；新的数据移动能力，如有效的传输验证；以及用于与对象存储通信的HTTP。由此产生的软件解决方案通过Globus Transfer、GridFTP和XSP开源软件发行版的组合交付给NSF社区。宇宙学、生物学、物理学、网络基础设施和其他领域的合作科学和工程项目正在协助评估。

Research progress often depends critically on the ability to move data over long distances: for example, from a remote supercomputer, telescope, or genome sequencing center to a researcher's laboratory or to a university data center. Yet despite major improvements in wide area and campus networks, data movement remains often frustratingly difficult and slow, due to obstacles such as firewalls and inadequate tools.To overcome this problem, this project develops an easy-to-use, high-performance data transfer and synchronization solution that can operate everywhere researchers need their data to be: desktops and research labs, campus servers and HPC facilities, supercomputing centers, scientific instruments, and public and private cloud providers. This new data movement solution builds on the widely used Globus implementation of the GridFTP protocol and the newer Globus Transfer hosted data movement service, plus the Indiana University implementation of the eXtensible Session Protocol (XSP), which provides for negotiation of end-to-end circuits via protocols such as OpenFlow. It incorporates numerous new capabilities, including negotiation of end-to-end circuits; communication between pairs of nodes that are both behind network address translation (NAT) devices or firewalls; new data movement capabilities such as efficient verification of transfers; and HTTP for communication with object stores.Resulting software solutions are delivered to the NSF community via a combination of Globus Transfer and the GridFTP and XSP open source software distributions. Partner science and engineering projects in cosmology, biology, physics, cyberinfrastructure, and other domains are assisting with evaluation.