CC* Integration-Small: Network cyberinfrastructure innovation with an intelligent real-time traffic analysis framework and application-aware networking

利用机器学习技术的智能分析方法为高吞吐量分布式计算工作流提供了分析、建模、预测和优化流量的新功能。这些技术可以通过访问来自边缘（校园网络）和核心（Internet 2）的真实数据以及即时（JIT）机器学习方法来大大增强。这种设计允许在校园网络基础设施上运行时部署模型，以做出实时网络决策。从这些网络基础设施收集的用于分析的网络流数据的规模迅速扩大，使得以现实和及时的方式执行网络流分析变得不可行。有内在的困难，源于数据存储，其格式和类型，以及在传统的分析方法来研究网络流数据。虽然过去几年在如何有效处理数据方面取得了进展，但新技术尚未很好地融入网络业务。需要通过利用现代数据存储格式和网络流数据的内在属性以及通过开发有效的数据结构和算法来改进网络流数据的分析方式。网络方面的最新进展允许由网络应用管理细粒度的网络控制策略。虽然有可能提高端到端科学数据传输的整体性能，但在实验/现场一级管理资源和区分网络服务方面存在问题。本项目的目标是通过JIT机器学习范式设计和开发智能网络分析，并通过可扩展的网络流分析框架进行加强，以实现高吞吐量计算框架中的网络应用感知控制。该项目通过与UNL的荷兰计算中心（HCC），开放科学联盟（OSG），阿贡国家实验室（ANL）和Internet 2的合作得到加强。该项目中开发的技术和框架将提供给开源社区，从而使研究和教育（R E）网络中的其他科学应用用例受益。该项目的重要目标是丰富UNL计算机学院学生的教育机会，并为更广泛的社区开展外展活动。该项目旨在通过以下方式改变当前的网络基础设施网络方法：（1）通过开发和整合机器学习的在线-离线方法来实时获得见解（与传统的离线方法不同），可以部署在数据中心进行实时网络流量分析和预测;（2）通过实现所开发的用于转换的理论模型，索引和构建搜索技术，以实时研究互联网规模的网络流数据，以及（3）通过应用感知软件定义网络（SDN）对数据传输的应用感知控制控制策略，为校园网络基础设施上的科学数据传输提供更大的网络管理和服务差异化灵活性。该奖项反映了NSF的法定使命，通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Intelligent analytics approaches leveraging machine learning techniques offer new capabilities to analyze, model, predict and optimize traffic for high-throughput distributed computing workflows. These techniques can be greatly enhanced by access to real-world data from the edge (campus networks) and the core (Internet2) as well as Just-In-Time (JIT) machine learning approaches. Such a design allows for run-time deployment of the models at the campus cyberinfrastructure to make real-time network decisions. Network flow data collected from these cyberinfrastructures for analysis quickly scales up in size, making it infeasible to perform analysis of network flows in a realistic and timely manner. There are intrinsic difficulties stemming from data storage, its formatting and types as well as the manner in which traditional analysis is done to study network flow data. Although advances have been made in the past several years in how data could be handled efficiently, the new techniques have not been integrated well into the network operations. Improvements need to be made in the way network flow data is analyzed by exploiting the modern data storage formats and the intrinsic properties of the network flow data as well as by developing efficient data structures and algorithms. Recent advances in networking allow for fine-grained network control policies to be managed by network applications. Although it is possible to improve the overall performance of scientific data transfers end-to-end, problems exist with managing resources and differentiating network services at the experiment/site level. Designing and developing intelligent network analysis by JIT machine learning paradigms strengthened by a scalable network flow analysis framework for an application-aware control of the network in high-throughput computing frameworks is the goal of this project. The project is strengthened by collaborations with Holland Computing Center (HCC) at UNL, Open Science Consortium (OSG), Argonne National Lab (ANL) and Internet2. The techniques and frameworks developed in this project will be made available to the open-source community, thus benefiting other science application use cases in Research and Education (R&E) networks. Enriching the education opportunities for UNL School of Computing students and conducting outreach events for the broader community are important objectives of this project.The project aims to transform the current cyberinfrastructure networking approach by (1) gaining insights in real-time by the development and integration of online-offline approaches to machine learning (unlike traditional offline approaches) that can be deployed in data centers for real-time network traffic analysis and prediction; (2) scalable analysis of network flow data by implementing the developed theoretical models for transforming, indexing and building search techniques to study the network flow data at internet-scale in real-time and (3) application-aware control of data transfers by application-aware software defined networking (SDN) control strategies to provide greater flexibility in network management and service differentiation for scientific data transfers on campus cyberinfrastructures.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.