Collaborative Research: CNS Core: Small: A New Architecture for Petabyte-scale File Transfer Evaluated in FABRIC

文件传输是互联网的基本操作。重要的科学仪器，如詹姆斯·韦伯太空望远镜和大型强子对撞机，每天都会产生大量文件。随着文件大小的增加，更有可能在其传输过程中引入错误，例如未被校验和进程捕获的错误。当文件存储捕捉罕见事件的数据时，可靠性尤其重要，例如罕见引力波，如果没有被发现，引力波的损失可能会严重影响科学结论。这个合作项目汇集了来自亚利桑那州立大学和波士顿大学的研究人员，以开发一个多层错误检测(MLED)架构。MLED体系结构中的每一层都由包括错误检测方法的范围等要求的策略来参数化。在聚合时，这些层提高了错误检测能力，并显著降低了未检测到错误的概率。该项目有三个研究方向：(1)开发mLED理论以研究策略参数之间的权衡，以优化未检测到的错误和文件传输延迟的概率；(2)设计利用Fabric中的功能的软件定义的mLED实现，这是一个新的网络计算系统试验台；(3)评估mLED，以调查和了解必须导航以利用Fabric的设计权衡，验证理论，并与文件传输解决方案的其他工具进行比较。该项目通过使用其他网络试验台所不具备的交换矩阵功能设计和实施mLED架构，对理解网络计算系统的科学产生了影响。当从数据得出的结论在很大程度上取决于它们的正确性时，可靠的文件传输尤其重要，这可能会对科学知识产生影响。此外，mLED推动了数据科学的发展，以更好地了解未检测到的错误、其通过文件传输的传播以及海量数据集的管理。交换矩阵跨国界提供了使用mLED进行PB级跨洋文件传输的机会。外联活动将与试验床社区分享使用Fabric的经验，并促进采用mLED进行文件传输。到这个项目结束时，它将为国家科学基金会和国家在网络计算系统方面的研究重点做出贡献，并帮助教育研究生和本科生，同时扩大代表不足的少数民族的参与。项目网站：https://www.public.asu.edu/~syrotiuk/mled.html.Jupyter笔记本将在整个项目中用于原型、探索和记录实验过程。为了实现可再现性，除所产生的出版物外，还将在项目网站上共享笔记本和代码库。本网站将在项目结束后保持至少三年。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

File transfer is a fundamental operation of the Internet. Important scientific instruments, such as the James Webb Space Telescope and the Large Hadron Collider, generate massive files daily. As files increase in size, it is more likely errors are introduced during their transfer, such as the ones not caught by the checksum process. Reliability is especially important when the files store data that capture a rare event, such as rare gravitational waves whose loss could significantly impact scientific conclusions if they go undetected. This collaborative project brings together investigators from Arizona State University and Boston University to develop a multi-layer error detection (MLED) architecture. MLED will not only eliminate detectable errors, it also provably reduces the probability of undetected errors.Each layer in the MLED architecture is parameterized by a policy that includes the scope of the error detection method among other requirements. When aggregated, the layers improve error detection capabilities and significantly reduce the probability of undetected errors. This project has three research directions: (1) to develop the theory of MLED to study trade-offs among policy parameters to optimize the probability of undetected errors and file transfer delay; (2) to design a software-defined implementation of MLED exploiting features in FABRIC, a new testbed of networked computing systems; (3) to evaluate MLED to investigate and understand the design trade-offs that must be navigated to harness FABRIC, to validate the theory, and to compare to other tools across the spectrum of file transfer solutions. This project impacts the science of understanding networked computing systems through the design and implementation of the MLED architecture using features of FABRIC that are unavailable in other network testbeds. Reliable file transfer is especially important when the conclusions drawn from the data critically depend on their correctness, with potential consequences to scientific knowledge. In addition, MLED advances data science to better understand undetected errors, their propagation through file transfer, and management of massive data sets. FABRIC Across Borders offers the opportunity to use MLED for petabyte-scale file transfer across the oceans. Outreach activities will share experience using FABRIC with the testbed community and promote the adoption of MLED for file transfer. By the end of this project, it will have contributed towards NSF and national research priorities on networked computing systems, and helped educate graduate and undergraduate students, while broadening participation of underrepresented minorities.Project website: https://www.public.asu.edu/~syrotiuk/mled.html. Jupyter Notebooks will be used throughout the project to prototype, explore, and document the experimental process. To enable reproducibility the notebooks and code repositories will be shared on the project website, in addition to resulting publications. This website will be maintained for at least three years following the project end date.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.