Reinforcement Learning for User Association and Handover in mmWave-enabled Networks
Reinforcement Learning for User Association and Handover in mmWave-enabled Networks
复制标题
DOI：
10.1109/twc.2022.3178767
复制
发表时间：
2022
影响因子：
10.4
通讯作者：
A. Alizadeh;M. Vu
中科院分区：
文献类型：
--
作者：
A. Alizadeh;M. Vu
我们运用多臂老虎机技术，针对毫米波网络中的负载均衡用户关联与切换，提出了集中式和半分布式在线算法。所有基站（BS）的负载均衡带来了明确的限制条件，使得所有用户设备（UE）的行为相互依赖，这给强化学习带来了颇具挑战性的难题。我们提出一种中央负载均衡器，以确保在每个学习步骤中所有基站都实现负载均衡。我们考虑两种关联向量：一种用于学习更新，另一种是截至目前最优的用于数据传输，这使得用户设备能够在无限期有效地参与后台学习过程的同时，进行产生最佳结果的数据传输。对于动态网络，我们引入一种测量模型，用以捕捉快速的信道变化和用户移动性。为了将切换率降至最低，我们还区分了传输和学习的切换成本，并引入一种随停留时间降低的学习切换成本。所提出的算法无需离线训练，可在线实施，且能有效适应网络动态变化。数值结果表明，所提算法学习收敛速度快，性能优于3GPP切换算法，在显著更高的网络总速率下，切换率降低一个数量级，达到接近最优的最差连接交换基准算法性能的94% - 97%。
Using a multi-armed bandit technique, we propose centralized and semi-distributed online algorithms for load balancing user association and handover in mmWave-enabled networks. Load balancing at all base stations (BSs) imposes explicit constraints that makes the actions of all user equipment (UEs) co-dependent, a challenging twist to reinforcement learning. We propose a central load balancer to guarantee load balancing at all BSs for every learning step. We consider two association vectors: one for leaning update, and one best-to-date for data transmission, allowing UEs to engage in best-result data transmission while effectively participating in a background learning process indefinitely. For dynamic networks, we introduce a measurement model capturing rapid channel variations and user mobility. To minimize handover rate, we also differentiate between handover costs for transmission and for learning, and introduce a learning handover cost decreasing with sojourn time. The proposed algorithms can be implemented online as they require no offline training and can effectively adapt to network dynamics. Numerical results show that the proposed algorithms exhibit fast learning convergence and outperform 3GPP handover by achieving an order of magnitude lower handover rate at a significantly higher network sum-rate, reaching within 94-97% of the near-optimal worst connection swapping benchmark algorithm.