CRII: OAC: Online Optimization of End-to-End Data Transfers in High Performance Networks

随着计算和传感技术的进步，科学应用产生的数据量迅速增长。为了适应这种增长，已经建立了容量高达400 Gbps的高速网络。尽管高速广域网络的可用性越来越高，并使用了为高性能而设计的现代数据传输协议，但实际上文件传输只达到了理论最大吞吐量的一小部分，导致网络未得到充分利用，用户也不满意。该项目旨在开发一种实时传输调优算法，以优化高速网络中的文件传输吞吐量。改进的数据传输性能不仅使分布式科学应用程序能够高效执行，而且通过减少共享数据所需的时间，还促进了位于不同地理位置的机构的科学家之间的合作。该项目通过提供最大限度地提高利用率的新颖解决方案，为构建下一代网络基础设施的努力提供了补充。该项目还促进了内华达大学里诺分校研究生级别的高性能网络课程的开发，并有助于本科生、女性学生和代表性不足学生的教育。因此，这项研究与NSF促进科学进步、促进国家繁荣和福利的使命是一致的。充分利用可用网络带宽以满足分布式科学工作流严格的端到端性能要求至关重要。然而，现有的数据传输应用(例如，SCP、BBCP和ftp)由于诸如终端系统限制、设计不当的传输协议和较差的存储性能等几个因素而无法饱和可用网络带宽。应用程序层传输调整提供了一个全面的解决方案，可显著提高传输吞吐量，并且只需在客户端进行修改即可应用。然而，由于搜索空间大以及网络和存储子系统的复杂动态，寻找应用层参数的最优配置是具有挑战性的。该项目将最先进的在线凸优化应用到应用层参数调整问题中，因为它即使在完全不确定的情况下也能提供性能和收敛保证。与博弈论启发的效用函数相结合，在线学习算法除了快速和最优外，还可以保证资源在用户之间的公平分配。该项目旨在通过异常检测和缓解来提高大型流媒体应用在动态网络条件下的性能。它有三个独特和创新的方面：(I)它使用最先进的在线学习算法来实时微调应用层参数。(Ii)它提高了样本传输的准确性和效率，最大限度地减少了实时调整的开销。(Iii)通过持续的性能监控和自适应调整，为对延迟敏感的传输(例如，高速流媒体应用)提供服务质量。该项目由高级网络基础设施办公室(OAC)和既定的激励竞争研究计划(EPSCoR)联合资助。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

With the advancement of computing and sensing technology, the amount of data generated by scientific applications is growing rapidly. To accommodate this growth, high speed networks with up to 400 Gbps capacities have been established. Despite the increasing availability of high-speed wide-area networks and the use of modern data transfer protocols designed for high performance, file transfers in practice attain only a fraction of theoretical maximum throughput, leaving networks underutilized and users unsatisfied. This project aims to develop a real-time transfer tuning algorithm to optimize file transfer throughput in high speed networks. Improved data transfer performance does not only enable efficient execution of distributed scientific applications but also fosters collaboration between scientists at geographically separated institutions by reducing time it takes to share data. This project complements the efforts to build next generation networking infrastructure by offering a novel solution to maximize utilization. The project also facilitates the development of a graduate level high-performance networking course at University of Nevada, Reno, and contribute to the education of undergraduate, female, and under-representative students. Therefore, this research aligns with the NSF's mission to promote the progress of science and to advance national prosperity, and welfare. It is critical to fully utilize available network bandwidth to meet stringent end-to-end performance requirements of distributed scientific workflows. Yet, existing data transfer applications (e.g., scp, bbcp, and ftp) fail to saturate the available network bandwidth due to several factors, such as end system limitations, ill-designed transfer protocols, and poor storage performances. Application-layer transfer tuning offers a comprehensive solution to enhance transfer throughput significantly and can be applied with only client-side modifications. However, finding optimal configuration for application-layer parameters is challenging due to large search space and complex dynamics of network and storage subsystems. This project applies state-of-the-art online convex optimization to application-layer parameter tuning problem as it offers performance and convergence guarantees even under complete uncertainty. In addition to being fast and optimal, online learning algorithms can guarantee the fair distribution of resources among users when combined with game-theory inspired utility functions. The project aims to improve the performance of large streaming applications under dynamic network conditions through anomaly detection and mitigation. It has three unique and innovative aspects: (i) It uses state-of-the art online learning algorithm to fine tune application-layer parameters in real-time. (ii) It improves accuracy and efficiency of sample transfers to minimize the overhead of real-time tuning. (iii) It offers quality of service for delay-sensitive transfers (e.g., high-speed streaming applications) through continuous performance monitoring and adaptive tuning.This project is jointly funded by Office of Advanced Cyberinfrastructure (OAC) and the Established Program to Stimulate Competitive Research (EPSCoR).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.