Collaborative Research: OAC Core: Small: Anomaly Detection and Performance Optimization for End-to-End Data Transfers at Scale

协作研究：OAC 核心：小型：大规模端到端数据传输的异常检测和性能优化

• 批准号: 2007829
• 负责人: Tevfik Kosar
• 金额: $ 22.5万
• 依托单位: SUNY at Buffalo
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-08-01 至 2024-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2007829&HistoricalAwards=false
• 关键词: Collaborative; Research; OAC; Core; Small

Despite continuous efforts and investments to upgrade the networking infrastructure of research and education institutions to meet the needs of large-scale science applications, the data transfers on these networks often perform very poorly. Understanding the underlying reasons for poor transfer performance is important yet challenging due to the sophisticated design of today's cyberinfrastructures. This project offers a set of novel models and algorithms to detect and mitigate performance issues of data transfers in research networks. The proposed suite of tools helps researchers and system administrators to pinpoint the root cause of performance problems of data transfers so that necessary actions can be taken swiftly to minimize their impact on ongoing transfers. The project will also integrate the research into all levels of education, including science projects with K-12 students, development of new curriculum modules for graduate- and undergraduate-level courses, and summer workshops specifically for minority groups.Understanding the true underlying reasons for poor transfer performance is key to mitigating them and delivering the promised transfer speeds. However, the involvement of multiple end systems, dynamically changing background traffic, and the complexity of today's networking infrastructures turns it into a complicated and time-consuming process. This project develops a novel anomaly-detection and performance-optimization framework for end-to-end data transfers at scale. The framework helps to predict, understand, diagnose, and optimize wide-area file transfers in today's extreme-scale cyberinfrastructures. To achieve this goal, it derives deep-neural-network-based predictive models that can relate transfer settings to throughput. These models are then used to estimate the optimal configuration for new transfers. The framework also gathers performance metrics for end-system and network resources periodically to keep track of system utilization. When a transfer anomaly is detected, the collected metrics are fed into anomaly-classification models to identify the root causes. Once the underlying reasons of performance problems are identified, the framework launches a real-time optimization process to reconfigure the transfer settings such that the impact of anomalies can be alleviated.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

尽管不断努力和投资，以更新研究和教育机构的网络基础设施，以满足大规模科学应用的需要，这些网络上的数据传输往往表现很差。由于当今网络基础设施的复杂设计，了解传输性能不佳的根本原因很重要，但也具有挑战性。该项目提供了一套新颖的模型和算法，以检测和减轻研究网络中数据传输的性能问题。拟议的工具套件可帮助研究人员和系统管理员查明数据传输性能问题的根本原因，以便迅速采取必要的行动，尽量减少对正在进行的传输的影响。 该项目还将把研究融入各级教育，包括K-12学生的科学项目，为研究生和本科生课程开发新的课程模块，以及专门针对少数群体的暑期研讨会。了解迁移表现不佳的真正根本原因是缓解这些问题并实现承诺的迁移速度的关键。然而，涉及多个终端系统、动态变化的后台流量以及当今网络基础设施的复杂性使其成为一个复杂且耗时的过程。 该项目开发了一种新的异常检测和性能优化框架，用于大规模的端到端数据传输。该框架有助于预测，理解，诊断和优化当今极端规模的网络基础设施中的广域文件传输。为了实现这一目标，它导出了基于深度神经网络的预测模型，可以将传输设置与吞吐量联系起来。然后，这些模型用于估计新转移的最佳配置。该框架还定期收集终端系统和网络资源的性能指标，以跟踪系统利用率。当检测到传输异常时，将收集的指标馈送到异常分类模型中以识别根本原因。一旦确定了性能问题的根本原因，该框架将启动实时优化流程，重新配置传输设置，从而减轻异常的影响。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Energy-saving Cross-layer Optimization of Big Data Transfer Based on Historical Log Analysis

基于历史日志分析的大数据传输节能跨层优化

• DOI: 10.1109/icc42927.2021.9500693
• 发表时间: 2021
• 期刊: ICC 2021 - IEEE International Conference on Communications
• 作者: Rodolph, Lavone;Zulkar Nine, MD S;Di Tacchio, Luigi;Kosar, Tevfik
• 通讯作者: Kosar, Tevfik

Energy-Efficient Data Transfer Optimization via Decision-Tree Based Uncertainty Reduction

通过基于决策树的不确定性降低实现节能数据传输优化

• DOI: 10.1109/icccn54977.2022.9868866
• 发表时间: 2022
• 期刊: 2022 International Conference on Computer Communications and Networks (ICCCN
• 作者: Jamil, Hasibul;Rodolph, Lavone;Goldverg, Jacob;Kosar, Tevfik
• 通讯作者: Kosar, Tevfik

SMURF: Efficient and Scalable Metadata Access for Distributed Applications

• DOI: 10.1109/tpds.2022.3175596
• 发表时间: 2021-05
• 期刊: IEEE Transactions on Parallel and Distributed Systems
• 影响因子: 5.3
• 作者: Bing Zhang;T. Kosar

GreenABR: energy-aware adaptive bitrate streaming with deep reinforcement learning

• DOI: 10.1145/3524273.3528188
• 发表时间: 2022-06
• 期刊: Proceedings of the 13th ACM Multimedia Systems Conference
• 作者: B. Turkkan;Ting Dai;Adithya Raman;T. Kosar;Changyou Chen;Muhammed Fatih Bulut;J. Zola;Daby M. Sow

Qualitative analysis of the relationship between design smells and software engineering challenges

设计味道与软件工程挑战之间关系的定性分析

• 发表时间: 2022
• 期刊: Proceedings of the 2022 European Symposium on Software Engineering (ESSE
• 作者: Asif Imran;Tevfik Kosar
• 通讯作者: Tevfik Kosar