Collaborative Research: CC-NIE Integration: A Data Movement Solution for Next-Generation Campus Cyberinfrastructure

点击翻译按钮获取中文摘要

Research progress often depends critically on the ability to move data over long distances: for example, from a remote supercomputer, telescope, or genome sequencing center to a researcher's laboratory or to a university data center. Yet despite major improvements in wide area and campus networks, data movement remains often frustratingly difficult and slow, due to obstacles such as firewalls and inadequate tools.To overcome this problem, this project develops an easy-to-use, high-performance data transfer and synchronization solution that can operate everywhere researchers need their data to be: desktops and research labs, campus servers and HPC facilities, supercomputing centers, scientific instruments, and public and private cloud providers. This new data movement solution builds on the widely used Globus implementation of the GridFTP protocol and the newer Globus Transfer hosted data movement service, plus the Indiana University implementation of the eXtensible Session Protocol (XSP), which provides for negotiation of end-to-end circuits via protocols such as OpenFlow. It incorporates numerous new capabilities, including negotiation of end-to-end circuits; communication between pairs of nodes that are both behind network address translation (NAT) devices or firewalls; new data movement capabilities such as efficient verification of transfers; and HTTP for communication with object stores.Resulting software solutions are delivered to the NSF community via a combination of Globus Transfer and the GridFTP and XSP open source software distributions. Partner science and engineering projects in cosmology, biology, physics, cyberinfrastructure, and other domains are assisting with evaluation.