CRII: OAC: Online Optimization of End-to-End Data Transfers in High Performance Networks

CRII：OAC：高性能网络中端到端数据传输的在线优化

• 批准号: 1850353
• 负责人: Engin Arslan
• 金额: $ 17.43万
• 依托单位: Board of Regents, NSHE, obo University of Nevada, Reno
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2019
• 资助国家: 美国
• 起止时间: 2019-03-15 至 2022-02-28
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1850353&HistoricalAwards=false
• 关键词: CRII; OAC; Online; Optimization; End

With the advancement of computing and sensing technology, the amount of data generated by scientific applications is growing rapidly. To accommodate this growth, high speed networks with up to 400 Gbps capacities have been established. Despite the increasing availability of high-speed wide-area networks and the use of modern data transfer protocols designed for high performance, file transfers in practice attain only a fraction of theoretical maximum throughput, leaving networks underutilized and users unsatisfied. This project aims to develop a real-time transfer tuning algorithm to optimize file transfer throughput in high speed networks. Improved data transfer performance does not only enable efficient execution of distributed scientific applications but also fosters collaboration between scientists at geographically separated institutions by reducing time it takes to share data. This project complements the efforts to build next generation networking infrastructure by offering a novel solution to maximize utilization. The project also facilitates the development of a graduate level high-performance networking course at University of Nevada, Reno, and contribute to the education of undergraduate, female, and under-representative students. Therefore, this research aligns with the NSF's mission to promote the progress of science and to advance national prosperity, and welfare. It is critical to fully utilize available network bandwidth to meet stringent end-to-end performance requirements of distributed scientific workflows. Yet, existing data transfer applications (e.g., scp, bbcp, and ftp) fail to saturate the available network bandwidth due to several factors, such as end system limitations, ill-designed transfer protocols, and poor storage performances. Application-layer transfer tuning offers a comprehensive solution to enhance transfer throughput significantly and can be applied with only client-side modifications. However, finding optimal configuration for application-layer parameters is challenging due to large search space and complex dynamics of network and storage subsystems. This project applies state-of-the-art online convex optimization to application-layer parameter tuning problem as it offers performance and convergence guarantees even under complete uncertainty. In addition to being fast and optimal, online learning algorithms can guarantee the fair distribution of resources among users when combined with game-theory inspired utility functions. The project aims to improve the performance of large streaming applications under dynamic network conditions through anomaly detection and mitigation. It has three unique and innovative aspects: (i) It uses state-of-the art online learning algorithm to fine tune application-layer parameters in real-time. (ii) It improves accuracy and efficiency of sample transfers to minimize the overhead of real-time tuning. (iii) It offers quality of service for delay-sensitive transfers (e.g., high-speed streaming applications) through continuous performance monitoring and adaptive tuning.This project is jointly funded by Office of Advanced Cyberinfrastructure (OAC) and the Established Program to Stimulate Competitive Research (EPSCoR).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

随着计算和传感技术的进步，科学应用产生的数据量正在迅速增长。为了适应这种增长，已经建立了高达400 Gbps容量的高速网络。尽管高速广域网的可用性越来越高，并且使用了为高性能而设计的现代数据传输协议，但实际上文件传输仅达到理论最大吞吐量的一小部分，使得网络未得到充分利用，用户不满意。该项目旨在开发一种实时传输调优算法，以优化高速网络中的文件传输吞吐量。改进的数据传输性能不仅可以有效地执行分布式科学应用程序，还可以通过减少共享数据所需的时间来促进地理上分散的机构的科学家之间的协作。该项目通过提供一种新颖的解决方案来最大限度地提高利用率，从而补充了构建下一代网络基础设施的努力。该项目还促进了里诺的内华达州大学研究生水平的高性能网络课程的开发，并有助于本科生、女性和代表性不足的学生的教育。因此，这项研究符合NSF的使命，以促进科学的进步，促进国家的繁荣和福利。充分利用可用的网络带宽以满足分布式科学工作流严格的端到端性能要求至关重要。然而，现有的数据传输应用（例如，SCP、BBCP和FTP）由于几个因素，例如终端系统限制、设计不良的传输协议和较差的存储性能，不能使可用网络带宽饱和。应用层传输调优提供了一个全面的解决方案，可以显著提高传输吞吐量，并且只需在客户端进行修改即可应用。然而，寻找应用层参数的最佳配置是具有挑战性的，由于大的搜索空间和复杂的动态网络和存储子系统。该项目将最先进的在线凸优化应用于应用层参数调整问题，因为即使在完全不确定的情况下，它也能提供性能和收敛保证。除了快速和最优之外，在线学习算法还可以在与博弈论启发的效用函数相结合时保证资源在用户之间的公平分配。该项目旨在通过异常检测和缓解来提高动态网络条件下大型流媒体应用程序的性能。它有三个独特和创新的方面：（i）它使用最先进的在线学习算法来实时微调应用层参数。(ii)它提高了样本传输的准确性和效率，以最大限度地减少实时调整的开销。(iii)它为延迟敏感的传输提供服务质量（例如，该项目由高级网络基础设施办公室（OAC）和激励竞争研究既定计划（EPSCoR）共同资助。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Sample Transfer Optimization with Adaptive Deep Neural Network

• DOI: 10.1109/indis49552.2019.00013
• 发表时间: 2019-11
• 期刊: 2019 IEEE/ACM Innovating the Network for Data-Intensive Science (INDIS)
• 作者: Hemanta Sapkota;M. Arifuzzaman;Engin Arslan

Time Series Analysis for Efficient Sample Transfers

高效样品转移的时间序列分析

• DOI: 10.1145/3322798.3329256
• 发表时间: 2019
• 期刊: ACM Workshop on Systems and Network Telemetry and Analytics
• 作者: Sapkota, Hemanta;Pehlivan, Bahadir A.;Arslan, Engin
• 通讯作者: Arslan, Engin

Swift and Accurate End-to-End Throughput Measurements for High-Speed Networks

快速、准确的高速网络端到端吞吐量测量

• 发表时间: 2022
• 期刊: The Network Traffic Measurement and Analysis Conference
• 作者: Arifuzzaman, Md;Arslan, Engin
• 通讯作者: Arslan, Engin

Towards Generalizable Network Anomaly Detection Models

迈向可推广的网络异常检测模型

• DOI: 10.1109/lcn52139.2021.9525015
• 发表时间: 2021
• 期刊: IEEE Conference on Local Computer Networks (LCN
• 作者: Arifuzzaman, Md;Islam, Shafkat;Arslan, Engin
• 通讯作者: Arslan, Engin

RIVA: Robust Integrity Verification Algorithm for High-Speed File Transfers

• DOI: 10.1109/tpds.2020.2966616
• 发表时间: 2020-01
• 期刊: IEEE Transactions on Parallel and Distributed Systems
• 影响因子: 5.3
• 作者: Batyr Charyyev;Engin Arslan