CNS Core: Small: Closing the Reality Gap for Learning-Augmented Network Systems

点击翻译按钮获取中文摘要

Modern Internet applications rely on sophisticated algorithms and systems to share network resources and deliver high quality of experience to each user (e.g., fast loading of web pages and smooth high-resolution video streaming). For instance, a video streaming system monitors the current speed of a user's internet connection and constantly changes the video quality to ensure smooth streaming at a high video quality. A key challenge of these systems is to ensure desirable user experience under different network environments, including different network speeds and different levels of network bandwidth changes. With the recent advances in machine learning (which makes predictions from data without following explicit instructions), many industry operators and researchers are exploring a new approach that automatically trains these algorithms as machine-learning models. While these learning-based systems show good performance in network environments similar to those the algorithms are trained in, they often do not perform well in new real-world network environments. Therefore, as new learning-based systems are developed and deployed every year, improving their generalization has become increasingly pressing.The goal of this project is to create a reusable framework to enhance the generalization of learning-based network systems. It focuses on systems that use deep reinforcement learning (DRL), and to improve their generalization, it applies formal tools from the machine learning literature and makes them efficient and effective for network systems by leveraging the traditional rule-based heuristics in the networking literature. The insight is that compared to DRL policies, rule-based heuristics (though suboptimal in some workloads) are less sensitive to differences between real systems/workloads and the simulated training environments and are more trusted by network operators. The project has three synergistic research thrusts. (1) It explores the use of rule-based heuristics to identify an appropriate level of randomization that should be introduced to the simulation-based training, in order to make the simulator-trained policies perform well in the real world. (2) To allow the offline-trained policies to generalize to a large and diverse operational space, the project iteratively improves the trained policy by periodically promoting difficult, yet improvable environments indicated by the performance of rule-based heuristics. (3) To cope with environment drifts in real network systems, the project proposes to run a fail-safe rule-based logic to collect the feedback data and use it to re-train the DRL policy in an unbiased and data-efficient fashion.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.