CC* Integration-Small: Error Free File Transfer for Big Science

科学数据传输已经变得如此之大，以至于以前在互联网上罕见的传输错误正在导致一些科学数据传输被破坏。互联网的错误检查机制是在兆字节还是一个大文件的时候设计的。现在文件可以包含数tb。旧的错误检查机制正处于不堪重负的危险之中。该项目旨在为互联网找到新的错误检查机制，以安全、高效、无错误地传输未来的科学数据。这个项目解决了两个基本问题。首先，Internet的校验和和消息摘要太小（32位），可能无法适应当今的错误模式。一个鲜为人知的事实是，校验和可以（而且通常应该）设计为可靠地捕获特定的错误。一个好的校验和被设计用来防止它实际会遇到的错误。因此，这个项目的第一步是收集目前在互联网上发生的各种传输错误的信息，进行全面的研究。其次，今天的文件传输协议，如果他们发现一个文件在传输过程中已经损坏，只需丢弃文件并再次传输它。在文件非常庞大（数十tb甚至pb长）的世界中，这是一种巨大的浪费。相反，文件传输协议应该设法修复文件的损坏部分。当项目收集有关错误的数据时，它还将设计一个新的文件传输协议，该协议可以增量地验证和修复文件。该项目将在未来几十年提高互联网支持科学和商业大数据传输的能力。用户将能够放心地传输大文件，数据将在网络上准确有效地复制。这项工作将进一步推进NSF的国家网络基础设施生态系统蓝图，确保网络高效运行，向任何需要的人提供可靠的大数据副本。有关该项目的更多信息可访问：www.hipft.netThis该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，认为值得支持。

Scientific data transfers have gotten so large that previously rare transmission errors in the Internet are causing some scientific data transfers to be corrupted. The Internet's error checking mechanisms were designed at a time when a megabyte was a large file. Now files can contain terabytes. The old error checking mechanisms are in danger of being overwhelmed. This project seeks to find new error checking mechanisms for the Internet to safely move tomorrow's scientific data efficiently and without errors.This project addresses two fundamental issues. First, the Internet's checksums and message digests are too small (32-bits) and probably are poorly tuned to today's error patterns. It is a little-known fact that checksums can (and typically should) be designed to reliably catch specific errors. A good checksum is designed to protect against errors that it will actually encounter. So the first step in this project is to collect information about the kinds of transmission errors currently happening in the Internet for a comprehensive study. Second, today's file transfer protocols, if they find a file has been corrupted in transit, simply discard the file and transfer it again. In a world in which the file is huge (tens of terabytes or even petabytes long), that's a tremendous waste. Rather, the file transfer protocol should seek to repair the corrupted parts of the file. As the project collects data about errors, it will also design a new file transfer protocol that can incrementally verify and repair files.This project will improve the Internet's ability to support big data transfers, both for science and commerce, for decades to come. Users will be able to transfer big files with confidence that the data will be accurately and efficiently copied over the network. This work will further NSF's Blueprint for a National Cyberinfrastructure Ecosystem by ensuring a world in which networks work efficiently to deliver trustworthy copies of big data to anyone who needs it. Additional information on the project is available at: www.hipft.netThis award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.