Large Scale Data Transfer: Addressing a Critical Need with a Low Cost Internet2 Service

大规模数据传输：通过低成本 Internet2 服务满足关键需求

• 批准号: 1160878
• 负责人: Stephen Wolff
• 金额: $ 4.95万
• 依托单位: INTERNET2
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2012
• 资助国家: 美国
• 起止时间: 2012-03-01 至 2014-02-28
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1160878&HistoricalAwards=false
• 关键词: Large; Scale; Data; Transfer; Addressing

Since the inception of the Internet, the problem of sending files between users that were too large for most servers has plagued users. Often the simple process of getting the sender and recipient to set up accounts on a common service to allow exchange of their respective identities results delays in actually sending a file, as this information needs to be known before a file exchange can be initiated. Internet2 pursues in this work the creation of a general bulk data transfer service that will allow researchers to stage datasets in the 10MB-1TB size range for short periods of time. An initial data store of 100TB will be provided as a short-term repository, allowing robust implementation to meet the needs of the community as none of the space will be allocated long-term and will be available for re-use quickly. This work focuses on modest functional goals (solving only the problem of exchanging files, rather than their long term storage and sharing) while emphasizing a global reach to work across a large population with no advanced user planning or provisioning. The intellectual merit of this work lies in its unique solution to a problem affecting a broad scope of research communities, particularly those that deal with large file data transfers, and in the value it will offer through a solution that already enjoys international deployment. The existence of a simple but comprehensive solution that is independent of any community infrastructure will be a critical component of the overall R&E cyberinfrastructure. FileSender, the core software to accomplish this, has evolved into an international open-source community that is gaining significant traction. By providing a service based on FileSender in the U.S., this project will capitalize on and extend the potential for this to become a ubiquitous and common solution throughout the international research and education community.

自从互联网诞生以来，用户之间发送文件的问题就一直困扰着用户，因为大多数服务器无法处理这些文件。通常，让发送方和接收方在公共服务上设置帐户以允许交换各自身份的简单过程会导致实际发送文件的延迟，因为在启动文件交换之前需要知道这些信息。在这项工作中，Internet2致力于创建一种通用的批量数据传输服务，使研究人员能够在短时间内将10MB-1TB大小的数据集存储起来。最初的100TB数据存储将作为短期存储库提供，允许稳健的实施，以满足社区的需求，因为没有任何空间将被长期分配，并且可以快速重用。这项工作的重点是适度的功能目标（只解决交换文件的问题，而不是解决文件的长期存储和共享问题），同时强调在没有高级用户规划或供应的情况下，在大量人口中工作的全球范围。这项工作的智力价值在于它对一个影响广泛的研究社区的问题的独特解决方案，特别是那些处理大文件数据传输的问题，以及它将通过一个已经享有国际部署的解决方案提供的价值。独立于任何社区基础设施的简单而全面的解决方案将是整个研发和电子网络基础设施的关键组成部分。FileSender是实现这一目标的核心软件，它已经发展成为一个国际开源社区，并获得了巨大的吸引力。通过在美国提供基于FileSender的服务，该项目将利用并扩展其潜力，使其成为整个国际研究和教育界普遍存在的通用解决方案。