CC* Integration-Small: Error Free File Transfer for Big Science

科学数据传输变得如此之大，以至于以前互联网上罕见的传输错误正在导致一些科学数据传输被破坏。互联网的错误检查机制是在1兆字节是大文件的时候设计的。现在，文件可以包含太字节。旧的错误检查机制正面临不堪重负的危险。这个项目试图为互联网找到新的错误检查机制，以便安全、高效、无错误地传输明天的科学数据。这个项目解决了两个基本问题。首先，互联网的校验和和消息摘要太小(32位)，可能不能很好地适应当今的错误模式。一个鲜为人知的事实是，可以(通常应该)将校验和设计为可靠地捕获特定错误。一个好的校验和旨在防止它实际遇到的错误。因此，这个项目的第一步是收集目前在互联网上发生的各种传输错误的信息，以便进行全面的研究。其次，当今的文件传输协议，如果发现文件在传输过程中已损坏，只需丢弃该文件并重新传输即可。在一个文件非常大(几十TB甚至PB长)的世界里，这是一种巨大的浪费。相反，文件传输协议应该设法修复文件的损坏部分。随着该项目收集有关错误的数据，它还将设计一种新的文件传输协议，可以增量地验证和修复文件。该项目将在未来几十年提高互联网支持大数据传输的能力，无论是科学还是商业。用户将能够放心地传输大文件，相信数据将通过网络准确而高效地复制。这项工作将通过确保一个网络高效工作的世界，向任何需要大数据的人提供值得信赖的大数据副本，从而进一步推进NSF的国家网络基础设施生态系统蓝图。有关该项目的更多信息，请访问www.hift.net。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Scientific data transfers have gotten so large that previously rare transmission errors in the Internet are causing some scientific data transfers to be corrupted. The Internet's error checking mechanisms were designed at a time when a megabyte was a large file. Now files can contain terabytes. The old error checking mechanisms are in danger of being overwhelmed. This project seeks to find new error checking mechanisms for the Internet to safely move tomorrow's scientific data efficiently and without errors.This project addresses two fundamental issues. First, the Internet's checksums and message digests are too small (32-bits) and probably are poorly tuned to today's error patterns. It is a little-known fact that checksums can (and typically should) be designed to reliably catch specific errors. A good checksum is designed to protect against errors that it will actually encounter. So the first step in this project is to collect information about the kinds of transmission errors currently happening in the Internet for a comprehensive study. Second, today's file transfer protocols, if they find a file has been corrupted in transit, simply discard the file and transfer it again. In a world in which the file is huge (tens of terabytes or even petabytes long), that's a tremendous waste. Rather, the file transfer protocol should seek to repair the corrupted parts of the file. As the project collects data about errors, it will also design a new file transfer protocol that can incrementally verify and repair files.This project will improve the Internet's ability to support big data transfers, both for science and commerce, for decades to come. Users will be able to transfer big files with confidence that the data will be accurately and efficiently copied over the network. This work will further NSF's Blueprint for a National Cyberinfrastructure Ecosystem by ensuring a world in which networks work efficiently to deliver trustworthy copies of big data to anyone who needs it. Additional information on the project is available at: www.hipft.netThis award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.