Collaborative Research: CNS CORE: Small: RUI: Hierarchical Deep Reinforcement Learning for Routing in Mobile Wireless Networks

在移动的无线网络中使用多跳路由正变得越来越普遍，正如这些网络正变得越来越密集、动态和异构。然而，由于需要使路由行为无缝地适应空间多样和时间变化的网络条件，因此为移动的无线网络设计通用的多跳路由策略是具有挑战性的。使用手工制作的路由策略的替代方案是使用强化学习（RL）来自动学习自适应多跳路由策略。RL专注于智能代理的设计：RL代理与其环境交互以学习策略，即，在不同的环境状态下采取什么行动。通过使用像深度强化学习（DeepRL）中的深度神经网络（DNN）这样的函数近似来近似策略，RL代理可以学习从其训练经验推广到看不见的网络条件，并将学习的路由策略扩展到更大的网络。 PI将继续他们目前的做法，让代表性不足的群体参与研究，并将利用项目研究，通过博士后指导，课程开发和推广活动来促进教学和培训。该项目的目标是使用DeepRL为移动的无线网络开发一个通用的多跳路由策略，该策略具有可扩展性，可推广性和自适应性。 具体来说，该项目将构建一个新的路由框架，该框架使用分层DeepRL来设计一个选项层次结构，由多层路由决策组成，共同实现网络的总体目标。为了使相同的路由策略能够在不同的设备和看不见的网络场景中使用，该框架将使用与新型神经网络模型相结合的关系特征来处理移动性并执行特征估计。为了进一步增强可推广性，该框架将使用持续学习来确保针对最近看到的网络场景学习的路由行为不会主导学习的路由策略。开发的路由策略将使用仿真和实验测试平台进行彻底评估。通过使用分层DeepRL，该项目将在开发基于RL的路由策略方面向前迈出重要一步，并将促进为各种移动的无线网络开发自适应策略。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The use of multi-hop routing in mobile wireless networks is becoming more prevalent, just as these networks are becoming more dense, dynamic, and heterogeneous. Designing a universal multi-hop routing strategy for mobile wireless networks is challenging, however, due to the need to seamlessly adapt routing behavior to spatially diverse and temporally changing network conditions. An alternative to using hand-crafted routing strategies is to use Reinforcement Learning (RL) to learn adaptive multi-hop routing strategies automatically. RL focuses on the design of intelligent agents: an RL agent interacts with its environment to learn a policy, i.e., which actions to take in different environmental states. By using function approximation like deep neural networks (DNNs) as in deep reinforcement learning (DeepRL) to approximate the policy, the RL agent can learn to generalize from its training experience to unseen network conditions and scale the learned routing strategy to larger networks. The PIs will continue their current practice of involving under-represented groups in research, and will use the project research to promote teaching and training through postdoctoral mentoring, course development, and outreach activities.The goal of this project is to use DeepRL to develop a universal multi-hop routing strategy for mobile wireless networks that is scalable, generalizable, and adaptive. Specifically, this project will build a novel routing framework that uses hierarchical DeepRL to design an option hierarchy, comprised of multiple layers of routing decisions working together to achieve the overall goals of the network. To enable the same routing strategy to be used at different devices and in unseen network scenarios, the framework will use relational features combined with novel neural network models to handle mobility and perform feature estimation. To further enhance generalizability, the framework will use continual learning to ensure that the routing behaviors learned for more recently seen network scenarios do not dominate the learned routing policy. The developed routing strategies will be thoroughly evaluated using both simulation and experimental testbeds. Through the use of hierarchical DeepRL, this project will provide a significant step forward in developing RL-based routing strategies, and will facilitate development of adaptive strategies for a wide range of mobile wireless networks.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.