Collaborative Research: CNS Core: Small: A New Architecture for Petabyte-scale File Transfer Evaluated in FABRIC

文件传输是互联网的基本操作。重要的科学仪器，如詹姆斯韦伯太空望远镜和大型强子对撞机，每天都会产生大量的文件。随着文件大小的增加，更有可能在传输过程中引入错误，例如校验和过程未捕获的错误。当文件存储捕获罕见事件的数据时，可靠性尤其重要，例如罕见的引力波，如果它们未被发现，其损失可能会严重影响科学结论。该合作项目汇集了来自亚利桑那州立大学和波士顿大学的研究人员，以开发多层错误检测（MLED）架构。MLED不仅可以消除可检测到的错误，还可以降低未检测到错误的概率。MLED架构中的每一层都由一个策略参数化，该策略包括错误检测方法的范围以及其他要求。 当聚合时，这些层提高了错误检测能力，并显著降低了未检测到错误的概率。本项目主要有三个研究方向：（1）发展MLED理论，研究策略参数之间的权衡，以优化未检测错误概率和文件传输延迟：（2）利用FABRIC（一种新的网络计算系统测试平台）的特性，设计MLED的软件定义实现;（3）评估MLED以调查和理解必须导航到利用FABRIC的设计权衡，验证理论，并与文件传输解决方案范围内的其他工具进行比较。该项目通过使用在其他网络测试平台中不可用的FABRIC功能设计和实现MLED架构，影响理解网络计算系统的科学。当从数据中得出的结论严重依赖于其正确性时，可靠的文件传输尤为重要，这可能会对科学知识产生影响。此外，MLED推进了数据科学，以更好地了解未检测到的错误，它们通过文件传输的传播以及海量数据集的管理。FABRIC Across Borders提供了使用MLED进行PB级跨海洋文件传输的机会。外联活动将与测试平台社区分享使用FABRIC的经验，并促进采用MLED进行文件传输。到本项目结束时，它将为NSF和国家网络计算系统的研究重点做出贡献，并帮助教育研究生和本科生，同时扩大代表性不足的少数民族的参与。https://www.public.asu.edu/~syrotiuk/mled.html在整个项目中，将使用Xuyter Notebooks来制作原型，探索和记录实验过程。为了实现可重复性，笔记本和代码库将在项目网站上共享，以及最终的出版物。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

File transfer is a fundamental operation of the Internet. Important scientific instruments, such as the James Webb Space Telescope and the Large Hadron Collider, generate massive files daily. As files increase in size, it is more likely errors are introduced during their transfer, such as the ones not caught by the checksum process. Reliability is especially important when the files store data that capture a rare event, such as rare gravitational waves whose loss could significantly impact scientific conclusions if they go undetected. This collaborative project brings together investigators from Arizona State University and Boston University to develop a multi-layer error detection (MLED) architecture. MLED will not only eliminate detectable errors, it also provably reduces the probability of undetected errors.Each layer in the MLED architecture is parameterized by a policy that includes the scope of the error detection method among other requirements. When aggregated, the layers improve error detection capabilities and significantly reduce the probability of undetected errors. This project has three research directions: (1) to develop the theory of MLED to study trade-offs among policy parameters to optimize the probability of undetected errors and file transfer delay; (2) to design a software-defined implementation of MLED exploiting features in FABRIC, a new testbed of networked computing systems; (3) to evaluate MLED to investigate and understand the design trade-offs that must be navigated to harness FABRIC, to validate the theory, and to compare to other tools across the spectrum of file transfer solutions. This project impacts the science of understanding networked computing systems through the design and implementation of the MLED architecture using features of FABRIC that are unavailable in other network testbeds. Reliable file transfer is especially important when the conclusions drawn from the data critically depend on their correctness, with potential consequences to scientific knowledge. In addition, MLED advances data science to better understand undetected errors, their propagation through file transfer, and management of massive data sets. FABRIC Across Borders offers the opportunity to use MLED for petabyte-scale file transfer across the oceans. Outreach activities will share experience using FABRIC with the testbed community and promote the adoption of MLED for file transfer. By the end of this project, it will have contributed towards NSF and national research priorities on networked computing systems, and helped educate graduate and undergraduate students, while broadening participation of underrepresented minorities.Project website: https://www.public.asu.edu/~syrotiuk/mled.html. Jupyter Notebooks will be used throughout the project to prototype, explore, and document the experimental process. To enable reproducibility the notebooks and code repositories will be shared on the project website, in addition to resulting publications. This website will be maintained for at least three years following the project end date.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.