Collaborative Research: OAC Core: Small: Anomaly Detection and Performance Optimization for End-to-End Data Transfers at Scale

协作研究：OAC 核心：小型：大规模端到端数据传输的异常检测和性能优化

• 批准号: 2007789
• 负责人: Engin Arslan
• 金额: $ 27.5万
• 依托单位: Board of Regents, NSHE, obo University of Nevada, Reno
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-08-01 至 2024-01-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2007789&HistoricalAwards=false
• 关键词: Collaborative; Research; OAC; Core; Small

Despite continuous efforts and investments to upgrade the networking infrastructure of research and education institutions to meet the needs of large-scale science applications, the data transfers on these networks often perform very poorly. Understanding the underlying reasons for poor transfer performance is important yet challenging due to the sophisticated design of today's cyberinfrastructures. This project offers a set of novel models and algorithms to detect and mitigate performance issues of data transfers in research networks. The proposed suite of tools helps researchers and system administrators to pinpoint the root cause of performance problems of data transfers so that necessary actions can be taken swiftly to minimize their impact on ongoing transfers. The project will also integrate the research into all levels of education, including science projects with K-12 students, development of new curriculum modules for graduate- and undergraduate-level courses, and summer workshops specifically for minority groups.Understanding the true underlying reasons for poor transfer performance is key to mitigating them and delivering the promised transfer speeds. However, the involvement of multiple end systems, dynamically changing background traffic, and the complexity of today's networking infrastructures turns it into a complicated and time-consuming process. This project develops a novel anomaly-detection and performance-optimization framework for end-to-end data transfers at scale. The framework helps to predict, understand, diagnose, and optimize wide-area file transfers in today's extreme-scale cyberinfrastructures. To achieve this goal, it derives deep-neural-network-based predictive models that can relate transfer settings to throughput. These models are then used to estimate the optimal configuration for new transfers. The framework also gathers performance metrics for end-system and network resources periodically to keep track of system utilization. When a transfer anomaly is detected, the collected metrics are fed into anomaly-classification models to identify the root causes. Once the underlying reasons of performance problems are identified, the framework launches a real-time optimization process to reconfigure the transfer settings such that the impact of anomalies can be alleviated.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

尽管不断努力和投资升级研究和教育机构的网络基础设施以满足大规模科学应用的需求，但这些网络上的数据传输往往表现较差。由于当今网络基础设施的精致设计，了解转移绩效不佳的根本原因是重要但又具有挑战性的。该项目提供了一系列新颖的模型和算法，以检测和减轻研究网络中数据传输的性能问题。拟议的工具套件可帮助研究人员和系统管理员查明数据传输绩效问题的根本原因，以便可以迅速采取必要的措施来最大程度地减少其对正在进行的转移的影响。 该项目还将将研究整合到所有级别的教育中，包括与K-12学生的科学项目，开发用于研究生和本科课程的新课程模块以及专门针对少数群体的夏季研讨会。理解不良转移绩效的真实理由是使他们降低和交付承诺的转移速度的关键。但是，多个最终​​系统的参与，动态变化的背景流量以及当今网络基础架构的复杂性将其变成了一个复杂且耗时的过程。 该项目为端到端数据传输的新型异常检测和性能优化框架开发。该框架有助于预测，理解，诊断和优化当今极端的网络基础设施中的广域文件传输。为了实现这一目标，它得出了基于深神经网络的预测模型，这些模型可以将转移设置与吞吐量相关联。然后，这些模型用于估计新传输的最佳配置。该框架还会定期收集最终系统和网络资源的性能指标，以跟踪系统利用率。当检测到转移异常时，收集的指标被送入异常分类模型以识别根本原因。一旦确定了绩效问题的根本原因，该框架就会启动一个实时优化过程，以重新配置转移设置，以便可以减轻异常的影响。该奖项反映了NSF的法定任务，并被认为是值得通过基金会的知识分子和更广泛影响的评估来审查审查的审查标准，可以通过评估来获得支持。

项目成果

Reliable Wide-Area Data Transfers for Streaming Workflows

• DOI: 10.1109/tpds.2022.3158673
• 发表时间: 2022
• 期刊: IEEE Transactions on Parallel and Distributed Systems
• 影响因子: 5.3
• 作者: Hemanta Sapkota;Engin Arslan

Falcon: Fair and Efficient Online File Transfer Optimization

• DOI: 10.1109/tpds.2023.3282872
• 发表时间: 2023-08
• 期刊: IEEE Transactions on Parallel and Distributed Systems
• 影响因子: 5.3
• 作者: Md. Arifuzzaman;B. Bockelman;James Basney;Engin Arslan

Use Only What You Need: Judicious Parallelism For File Transfers in High Performance Networks

• DOI: 10.1145/3577193.3593722
• 发表时间: 2023-06
• 期刊: Proceedings of the 37th International Conference on Supercomputing
• 作者: Md. Arifuzzaman;Engin Arslan

Learning Transfers via Transfer Learning

通过迁移学习进行学习迁移

• 发表时间: 2021
• 期刊: Innovating the Network for Data-Intensive Science
• 作者: Arifuzzaman, MD;Arslan, Engin
• 通讯作者: Arslan, Engin

Avoiding data loss and corruption for file transfers with Fast Integrity Verification

• DOI: 10.1016/j.jpdc.2021.02.002
• 发表时间: 2021-02
• 期刊: J. Parallel Distributed Comput.
• 作者: Ahmed Alhussen;Engin Arslan