A Deep Reinforcement Learning Framework for Spectrum Management in Dynamic Spectrum Access
A Deep Reinforcement Learning Framework for Spectrum Management in Dynamic Spectrum Access
复制标题
DOI：
10.1109/jiot.2021.3052691
复制
发表时间：
2021-07-15
影响因子：
10.6
通讯作者：
Yi, Yang
中科院分区：
文献类型：
--
作者：
Song, Hao;Liu, Lingjia;Yi, Yang
动态频谱接入（DSA）在缓解频谱紧张、提高网络容量方面具有巨大的潜力.然而，必须解决两个基本的技术问题，即，DSA用户之间的干扰协调和主用户（PU）的干扰抑制。这两个问题是非常具有挑战性的，因为通常在DSA网络中没有强大的基础设施来支持集中式控制。因此，DSA用户必须单独执行频谱管理，包括频谱接入和功率分配，而没有准确的信道状态信息和集中控制。本文提出了一种新的频谱管理框架，该框架利用强化学习中的Q-学习，使DSA用户能够个性化、智能化地进行有效的频谱管理。为了更有效的过程，神经网络（NN）被用来实现Q学习过程，即所谓的深度Q网络（DQN）。此外，我们还研究了最佳的方式来构建DQN考虑无线通信的性能和神经网络训练的难度。最后，进行了广泛的仿真研究，以证明所提出的频谱管理框架的有效性。
Dynamic spectrum access (DSA) has the great potential to alleviate spectrum shortage and promote network capacity. However, two fundamental technical issues have to be addressed, namely, interference coordination between DSA users and interference suppression for primary users (PUs). These two issues are very challenging since generally there is no powerful infrastructures in DSA networks to support centralized control. As a result, DSA users have to perform spectrum management individually, including spectrum access and power allocation, without accurate channel state information and centralized control. In this article, a novel spectrum management framework is proposed, in which Q-learning, a type of reinforcement learning, is utilized to enable DSA users to carry out effective spectrum management individually and intelligently. For more efficient process, neural networks (NNs) are employed to implement Q-learning processes, so-called deep Q-network (DQN). Furthermore, we also investigate the optimal way to construct DQN considering both the performance of wireless communications and the difficulty of NN training. Finally, extensive simulation studies are conducted to demonstrate the effectiveness of the proposed spectrum management framework.