SDCI Net: UD* - A UDT-Based Application Suite for High Performance Data Transport

基于UDP的数据传输(UDT)是一种数据传输协议，可作为开源软件库使用。UDT旨在支持在广域高性能网络上传输大数据集。在这些情况下，TCP常常是无效的。有了UDT，用户可以通过10 Gb/S广域网以超过9 GB/S的速度将数据从磁盘发送到磁盘。此外，UDT还用于通过UPD穿孔在防火墙之间移动数据，以及用于维护大量连接。后者对于数据密集型计算等应用程序非常有用。在这两种情况下，TCP都不能很好地工作。UDT也是可配置的，因此用户可以插入适合专门网络拓扑或应用的定制控制算法。UD*项目的目标是使UDT成为科学数据传输的标准协议，并促进科学研究团体使用UDT。虽然已经提出了许多解决方案来解决TCP效率低下的问题，但今天许多用户在移动大型数据集时仍然遇到麻烦。UD*项目有三个组成部分。第一个组成部分是通过为UDT开发基于Web的前端，并将UDT集成到用于移动数据集的标准实用程序(如rsync)中，从而使UDT更易于访问。第二个组成部分是向UDT社区提供网络和软件工程支持，包括向感兴趣的用户提供技术援助、改进文档、回复邮件列表和博客中的查询、创建教程材料、举办讲习班和相关活动。UD*项目还为美国国家科学基金会支持的三个科学家社区提供直接技术支持：i)使用开放云联盟--S开放科学数据云的各种科学家；ii)使用Bio-Mirror的生物学家？S--各种镜像站点；iii)以及通过连接到芝加哥星光设施的40G和100G网络移动大型数据集的科学家。第三个组成部分是开发两个新版本的UDT。UDT5将包括支持数据密集型计算应用程序和数据中心规模计算应用程序的功能。UDX将扩展到40GE和100GE广域网络。科学仪器、传感器和其他设备的数量和产出能力都在以摩尔-S定律的速度增长。大数据集和高性能网络正变得越来越普遍，然而在广域高性能网络上传输大数据集仍然存在根本问题。这一问题可以通过建立和支持UDT社区并增强UDT来为新兴应用程序提供特定支持来部分解决，例如数据密集型计算应用程序和40G和100G广域网络上的应用程序。用于传输、存储、可视化和共享多TB数据集的技术对于包括气候建模、模拟和国土防御应用在内的大量科学和国防应用具有广泛的重要意义。UD*可以对任何需要处理非常大的数据集的学科产生直接的变革性影响。UD*项目开发教程，支持UDT用户组，并教授关于UD*的一天工作坊，以扩大可以使用UDT传输大型数据集的用户数量。

UDP-based Data Transfer (UDT) is a data transport protocol that is available as an open source software library. UDT was designed to support transferring large datasets over wide area high performance networks. TCP is often times ineffective in these situations. With UDT, users can send data from disk-to-disk at over 9Gb/s over a 10Gb/s wide area network. In addition, UDT has also been used for moving data across firewalls with UPD hole punching and for maintaining a very large number of connections. The latter is useful for applications such as data intensive computing. TCP does not work well in either of these situations. UDT is also configurable so that users can plug in customized control algorithms appropriate for specialized network topologies or applications.The goal of the UD* Project is to make UDT a standard protocol for scientific data transfer and to facilitate the use of UDT by the scientific research community. Although there have been many proposed solutions to address the TCP inefficiency problem, many users today still experience trouble when moving large datasets. The UD* project has three components.The first component is to make UDT more accessible by developing a web-based front end to UDT and by integrating UDT into standard utilities that are used for moving datasets, such as rsync. The second component is to provide network and software engineering support to the UDT Community, including providing technical assistance to interested users, improving documentation, responding to queries in mailing lists and blogs, creating tutorial materials, running workshops, and related activities. The UD* Project also provides direct technical support to three communities of NSF-supported scientists: i) the various scientists making use of the Open Cloud Consortium?s Open Science Data Cloud; ii) biologists using Bio-mirror?s various mirror sites; iii) and scientists moving large datasets over 40G and 100G networks that connect to the StarLight Facility in Chicago. The third component is to develop two new versions of UDT. UDT5 will include features to support data intensive computing applications and data center scale computing applications. UDX will be designed to scale to 40GE and 100GE wide area networks.Intellectual merit. The number and output capacity of scientific instruments, sensors and other devices are growing at the rate of Moore?s Law. Large datasets and high performance networks are becoming increasingly common, yet there are still fundamental problems transporting large datasets over wide area high performance networks. This problem can be addressed in part by building and supporting a UDT Community and enhancing UDT to provide specific support for emerging applications, such as data intensive computing applications and applications over 40G and 100G wide area networks.Broader Impact. Technology for transporting, storing, visualizing, and sharing multiple terabyte datasets is broadly important for a large number of scientific and defense applications, including climate modeling, simulation, and homeland defense applications. UD* can have a direct transformative impact on any discipline that requires working with very large datasets. The UD* Project develops tutorials, supports a UDT Users Group, and teaches one day workshops on UD* to broaden the number of users that can use UDT for transporting large datasets.