COREL: Constrained Reinforcement Learning for Video Streaming ABR Algorithm Design Over mmWave 5G
COREL: Constrained Reinforcement Learning for Video Streaming ABR Algorithm Design Over mmWave 5G
复制标题
DOI：
10.1109/cqr59928.2023.10317803
复制
发表时间：
2023-10
期刊：
影响因子：
--
通讯作者：
Xinyue Hu;Arnob Ghosh;Xin Liu;Zhi-Li Zhang;N. Shroff
中科院分区：
文献类型：
--
作者：
Xinyue Hu;Arnob Ghosh;Xin Liu;Zhi-Li Zhang;N. Shroff
自适应比特率选择（ABR）机制，它决定每个视频块的比特率是视频流的重要组成部分。由于它们能够根据过去的数据学习有效的比特率动作，并且它们对有线，3G和4G网络的改善，因此对开发基于增强的学习（RL）ABR算法的增强算法引起了重大兴趣。但是，由于广泛而疯狂的波动吞吐量，经验质量（QOE）（QOE），包括基于RL的方法的最新ABR算法（包括基于RL的方法）的最新ABR算法的期望不足。这些算法找到了多目标无约束问题的最佳策略，其中策略固有地依赖于多个目标的预定义权重参数（例如，比特率最大化，失速时间最小化）。我们的经验评估表明，这样的政策不能充分适应5G吞吐量的高变化，从而导致了长时间的失速时间。为了解决这些问题，我们将ABR选择问题作为约束的马尔可夫决策过程，目的是最大化QoE受摊位时间约束。该公式的强度是，它有助于减少失速时间，同时保持高比特率。我们提出了一种原始的双偶会批判性RL算法Corel，该算法与现有基于RL的方法相比，它结合了一个额外的评论家网络以估算失速时间，并可以调整最佳的双重变量或重量，以指导政策以最大程度地减少失速时间。我们在各种商业MMWAVE 5G痕迹上的实验结果表明，Corel将平均失速时间降低了4倍，而第95个百分位数则减少了2倍。
The adaptive bitrate selection (ABR) mechanism, which decides the bitrate for each video chunk is an important part of video streaming. There has been significant interest in developing Reinforcement-Learning (RL) based ABR algorithms because of their ability to learn efficient bitrate actions based on past data and their demonstrated improvements over wired, 3G and 4G networks. However, the Quality of Experience (QoE), especially video stall time, of state-of-the-art ABR algorithms including the RL-based approaches falls short of expectations over commercial mmWave 5G networks, due to widely and wildly fluctuating throughput. These algorithms find optimal policies for a multi-objective unconstrained problem where the policies inherently depend on the predefined weight parameters of the multiple objectives (e.g., bitrate maximization, stall-time minimization). Our empirical evaluation suggests that such a policy cannot adequately adapt to the high variations of 5G throughput, resulting in long stall times. To address these issues, we formulate the ABR selection problem as a constrained Markov Decision Process where the objective is to maximize the QoE subject to a stall-time constraint. The strength of this formulation is that it helps mitigate the stall time while maintaining high bitrates. We propose COREL, a primal-dual actor-critic RL algorithm, which incorporates an additional critic network to estimate stall time compared to existing RL-based approaches and can tune the optimal dual variable or weight to guide the policy towards minimizing stall time. Our experiment results across various commercial mmWave 5G traces reveal that COREL reduces the average stall time by a factor of 4 and the 95th percentile by a factor of 2.