CC* Integration-Small: Network cyberinfrastructure innovation with an intelligent real-time traffic analysis framework and application-aware networking

利用机器学习技术的智能分析方法为高吞吐量分布式计算工作流提供了分析、建模、预测和优化流量的新功能。通过从边缘(园区网络)和核心(Internet2)访问真实世界数据以及即时(JIT)机器学习方法，可以极大地增强这些技术。这样的设计允许在园区网络基础设施中运行时部署模型，以做出实时网络决策。从这些网络基础设施收集的网络流量数据用于分析的规模迅速扩大，使得无法现实和及时地执行网络流量分析。数据存储、其格式和类型以及进行传统分析以研究网络流量数据的方式存在固有的困难。尽管在过去几年中，在如何有效处理数据方面取得了进展，但新技术还没有很好地整合到网络操作中。需要通过利用现代数据存储格式和网络流数据的内在属性以及开发高效的数据结构和算法来改进分析网络流数据的方式。网络方面的最新进展允许由网络应用程序管理细粒度的网络控制策略。尽管端到端地提高科学数据传输的整体性能是可能的，但在实验/站点级别上管理资源和区分网络服务存在问题。本项目的目标是通过JIT机器学习范例设计和开发智能网络分析，并通过可扩展的网络流分析框架在高吞吐量计算框架中实现对网络的应用感知控制。该项目通过与联合国大学荷兰计算中心、开放科学联盟、阿贡国家实验室和Internet2的合作得到加强。在这个项目中开发的技术和框架将提供给开源社区，从而使研究和教育(R&A；E)网络中的其他科学应用用例受益。这个项目的重要目标是丰富UNL计算学院学生的教育机会，并为更广泛的社区开展外联活动。该项目旨在通过以下方式改变当前的网络基础设施联网方法：(1)通过开发和集成线上-线下机器学习方法(不同于传统的离线方法)实时获得洞察力，这些方法可以部署在数据中心进行实时网络流量分析和预测；(2)通过实施开发的转换、索引和构建搜索技术的理论模型对网络流量数据进行可扩展分析，以实时研究互联网规模的网络流量数据；(3)通过应用感知软件定义的网络(SDN)控制策略对数据传输进行应用感知控制，为校园网络基础设施上的科学数据传输在网络管理和服务差异化方面提供更大的灵活性。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Intelligent analytics approaches leveraging machine learning techniques offer new capabilities to analyze, model, predict and optimize traffic for high-throughput distributed computing workflows. These techniques can be greatly enhanced by access to real-world data from the edge (campus networks) and the core (Internet2) as well as Just-In-Time (JIT) machine learning approaches. Such a design allows for run-time deployment of the models at the campus cyberinfrastructure to make real-time network decisions. Network flow data collected from these cyberinfrastructures for analysis quickly scales up in size, making it infeasible to perform analysis of network flows in a realistic and timely manner. There are intrinsic difficulties stemming from data storage, its formatting and types as well as the manner in which traditional analysis is done to study network flow data. Although advances have been made in the past several years in how data could be handled efficiently, the new techniques have not been integrated well into the network operations. Improvements need to be made in the way network flow data is analyzed by exploiting the modern data storage formats and the intrinsic properties of the network flow data as well as by developing efficient data structures and algorithms. Recent advances in networking allow for fine-grained network control policies to be managed by network applications. Although it is possible to improve the overall performance of scientific data transfers end-to-end, problems exist with managing resources and differentiating network services at the experiment/site level. Designing and developing intelligent network analysis by JIT machine learning paradigms strengthened by a scalable network flow analysis framework for an application-aware control of the network in high-throughput computing frameworks is the goal of this project. The project is strengthened by collaborations with Holland Computing Center (HCC) at UNL, Open Science Consortium (OSG), Argonne National Lab (ANL) and Internet2. The techniques and frameworks developed in this project will be made available to the open-source community, thus benefiting other science application use cases in Research and Education (R&E) networks. Enriching the education opportunities for UNL School of Computing students and conducting outreach events for the broader community are important objectives of this project.The project aims to transform the current cyberinfrastructure networking approach by (1) gaining insights in real-time by the development and integration of online-offline approaches to machine learning (unlike traditional offline approaches) that can be deployed in data centers for real-time network traffic analysis and prediction; (2) scalable analysis of network flow data by implementing the developed theoretical models for transforming, indexing and building search techniques to study the network flow data at internet-scale in real-time and (3) application-aware control of data transfers by application-aware software defined networking (SDN) control strategies to provide greater flexibility in network management and service differentiation for scientific data transfers on campus cyberinfrastructures.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.