CC* Integration-Large: Q-Factor: A Framework to Enable Ultra High-Speed Data Transfer Optimization based on Real-Time Network State Information provided by Programmable Data Planes

通信网络是当今科学工作流程的关键组成部分。研究人员需要长距离、超高速的网络来传输来自采集站点的大量数据（例如Vera C。鲁宾天文台，也被称为智利的大型综合巡天望远镜），以处理网站，并与世界各地的科学家分享测量结果。然而，虽然网络带宽不断增加，但由于网络传输协议的参数设置的固有限制以及终端主机处缺乏网络状态信息，大多数数据传输无法有效地利用增加的容量。为了应对这些挑战，Q-Factor计划使用临时网络状态数据来动态配置当前传输协议参数，以达到更高的网络利用率，从而改善科学工作流程。Q-Factor利用可编程网络设备和带内网络遥测（INT）应用，并提供软件解决方案来处理终端主机的带内测量。在数据传输节点（DTN）上使用Q因子，TCP/IP参数将根据时间网络特性（例如往返时间、网络利用率和网络拥塞）进行配置。这种调整预计将提高网络利用率，缩短流完成时间，并显著减少由网络缓冲区溢出引起的数据包丢失。此外，Q-Factor还通过将内核参数和缓冲区调整到最佳大小来节省主机内存。Q-Factor针对通信网络中的一个及时问题：科学工作流程中超高速网络的利用率不足。为了保持科学进步不受限制，未来的科学工作流程需要支持新兴的数据密集型科学实验（例如，Vera Rubin天文台，高亮度大型强子对撞机），其中数据生成显著增长，每年达到艾字节的流量。该项目的结果还将使我们更好地了解网络设备在巨大流量下的最佳缓冲区大小以及各种拥塞控制算法的相互作用。实验测量数据、网络状态信息、网络拓扑、软件代码、TCP调整指南和结果将在Q-Factor网站www.example.com上提供，https://q-factor.io该网站将在项目完成后至少三年内进行维护和索引。这将有助于我们更好地了解网络设备的最佳缓冲区大小。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Communication networks are critical components of today’s scientific workflows. Researchers require long-distance, ultra high-speed networks to transfer huge data from acquisition sites (such as Vera C. Rubin Observatory, also knowns as Large Synoptic Survey Telescope in Chile) to processing sites, and to share measurements with scientists worldwide. However, while network bandwidth is continuously increasing, the majority of data transfers are unable to efficiently utilize the added capacity due to inherent limitations of parameter settings of the network transport protocols and the lack of network state information at the end hosts. To address these challenges, Q-Factor plans to use temporal network state data to dynamically configure current transport protocol parameters to reach higher network utilization and, as a result, to improve scientific workflows.Q-Factor leverages programmable network devices with the In-band Network Telemetry (INT) application and delivers a software solution to process in-band measurements at the end hosts. Using Q-Factor on Data Transfer Nodes (DTN)s, TCP/IP parameters will be configured according to temporal network characteristics, such as round-trip time, network utilization, and network congestion. This tuning is expected to result in increased network utilization, shorter flow completion times, and significantly fewer packet drops caused by network buffers overflow. Additionally, Q-Factor is geared to save host memory by tailoring kernel parameters and buffers to optimal sizes.Q-Factor targets a timely issue in communication networks: underutilization of ultra high-speed networks for science workflows. In order to keep scientific progress unconstrained, future science workflows need to support emerging data-intensive science experiments (e.g., the Vera Rubin Observatory, High Luminosity Large Hadron Collider) where data generation grows significantly, reaching exabytes of traffic each year. Results of this project will also allow better understanding of optimal buffer sizes of network devices for huge flows and the interaction of various congestion control algorithms.Experimental measurement data, network state information, network topology, software code, TCP tuning guidelines, and results will be available on the Q-Factor website https://q-factor.io, which will be maintained and indexed for at least three years after the completion of the project.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.