CNS Core: Small: Closing the Reality Gap for Learning-Augmented Network Systems

现代互联网应用依赖于复杂的算法和系统来共享网络资源并为每个用户提供高质量的体验(例如，快速加载网页和流畅的高分辨率视频流)。例如，视频流系统监控用户互联网连接的当前速度，并不断改变视频质量，以确保以高视频质量流畅地流传输。这些系统的一个关键挑战是确保在不同的网络环境下获得理想的用户体验，包括不同的网络速度和不同级别的网络带宽变化。随着机器学习的最新进展(不需要遵循明确的指令就能根据数据做出预测)，许多行业运营商和研究人员正在探索一种新的方法，将这些算法自动训练为机器学习模型。虽然这些基于学习的系统在类似于算法训练的网络环境中表现出良好的性能，但它们在新的真实网络环境中往往表现不佳。因此，随着每年新的基于学习的系统的开发和部署，提高其普适性变得越来越紧迫。本项目的目标是创建一个可复用的框架，以增强基于学习的网络系统的普适性。它专注于使用深度强化学习(DRL)的系统，为了提高它们的泛化，它应用了机器学习文献中的形式化工具，并通过利用网络文献中传统的基于规则的启发式方法，使它们对网络系统高效有效。与DRL策略相比，基于规则的启发式方法(尽管在某些工作负载中不是最优的)对实际系统/工作负载和模拟培训环境之间的差异不那么敏感，并且更受网络运营商的信任。该项目有三个协同研究推动力。(1)探索了使用基于规则的启发式方法来确定应该引入到基于模拟的训练中的适当的随机化水平，以便使模拟器训练的策略在真实世界中表现良好。(2)为了允许离线训练的策略推广到大的和多样化的操作空间，该项目通过周期性地促进困难但可改善的环境来迭代地改进训练的策略，该环境由基于规则的启发式算法的性能指示。(3)为了应对真实网络系统中的环境漂移，该项目建议运行故障安全规则逻辑来收集反馈数据，并使用它以不偏不倚和数据高效的方式重新训练DRL政策。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Modern Internet applications rely on sophisticated algorithms and systems to share network resources and deliver high quality of experience to each user (e.g., fast loading of web pages and smooth high-resolution video streaming). For instance, a video streaming system monitors the current speed of a user's internet connection and constantly changes the video quality to ensure smooth streaming at a high video quality. A key challenge of these systems is to ensure desirable user experience under different network environments, including different network speeds and different levels of network bandwidth changes. With the recent advances in machine learning (which makes predictions from data without following explicit instructions), many industry operators and researchers are exploring a new approach that automatically trains these algorithms as machine-learning models. While these learning-based systems show good performance in network environments similar to those the algorithms are trained in, they often do not perform well in new real-world network environments. Therefore, as new learning-based systems are developed and deployed every year, improving their generalization has become increasingly pressing.The goal of this project is to create a reusable framework to enhance the generalization of learning-based network systems. It focuses on systems that use deep reinforcement learning (DRL), and to improve their generalization, it applies formal tools from the machine learning literature and makes them efficient and effective for network systems by leveraging the traditional rule-based heuristics in the networking literature. The insight is that compared to DRL policies, rule-based heuristics (though suboptimal in some workloads) are less sensitive to differences between real systems/workloads and the simulated training environments and are more trusted by network operators. The project has three synergistic research thrusts. (1) It explores the use of rule-based heuristics to identify an appropriate level of randomization that should be introduced to the simulation-based training, in order to make the simulator-trained policies perform well in the real world. (2) To allow the offline-trained policies to generalize to a large and diverse operational space, the project iteratively improves the trained policy by periodically promoting difficult, yet improvable environments indicated by the performance of rule-based heuristics. (3) To cope with environment drifts in real network systems, the project proposes to run a fail-safe rule-based logic to collect the feedback data and use it to re-train the DRL policy in an unbiased and data-efficient fashion.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.