Graph Learning Based Head Movement Prediction for Interactive 360 Video Streaming
Graph Learning Based Head Movement Prediction for Interactive 360 Video Streaming
复制标题
基于图学习的交互式 360 度视频流头部运动预测
DOI：
10.1109/tip.2021.3073283
复制
发表时间：
2021-04
影响因子：
10.6
通讯作者：
Tan Jack Z. G.
中科院分区：
文献类型：
--
作者：
Zhang Xue;Cheung Gene;Zhao Yao;Le Callet Patrick;Lin Chunyu;Tan Jack Z. G.
以高质量编码的超高清(UHD)360视频通常太大，无法通过带宽(BW)受限的网络进行完整的流媒体传输。一种流行的方法是在头盔显示器(HMD)中交互地提取并发送对应于观看者的当前视场(FOV)的空间子区域，以获得更高BW效率的流传输。由于服务器和客户端之间存在不可忽视的往返时间(RTT)延迟，准确的头部运动预测预测观众未来的视场是至关重要的。本文将头部运动预测问题归结为一个稀疏有向图学习问题：将采集到的观众头部运动轨迹、360图像显著图和生物人头模型三个相关信息源提炼成一个视图转移马尔可夫模型。具体地说，我们用三个信息源定义了一个带有似然和先验项的约束最大后验概率(MAP)问题。我们交替使用迭代重加权最小二乘(IRLS)和Frank-Wolfe(FW)优化策略来求解MAP问题。在每一次FW迭代中，求解一个线性规划(LP)，其运行时间由于热启动初始化而减少。在根据数据估计了马尔可夫模型之后，我们使用它来优化一个基于瓦片的360视频流媒体系统。大量的实验表明，我们的头部运动预测方案的性能明显优于现有的方案，我们优化的基于瓦片的流媒体方案在率失真性能上也优于竞争对手。
Ultra-high definition (UHD) 360 videos encoded in fine quality are typically too large to stream in its entirety over bandwidth (BW)-constrained networks. One popular approach is to interactively extract and send a spatial sub-region corresponding to a viewer’s current field-of-view (FoV) in a head-mounted display (HMD) for more BW-efficient streaming. Due to the non-negligible round-trip-time (RTT) delay between server and client, accurate head movement prediction foretelling a viewer’s future FoVs is essential. In this paper, we cast the head movement prediction task as a sparse directed graph learning problem: three sources of relevant information—collected viewers’ head movement traces, a 360 image saliency map, and a biological human head model—are distilled into a view transition Markov model. Specifically, we formulate a constrained maximum a posteriori (MAP) problem with likelihood and prior terms defined using the three information sources. We solve the MAP problem alternately using a hybrid iterative reweighted least square (IRLS) and Frank-Wolfe (FW) optimization strategy. In each FW iteration, a linear program (LP) is solved, whose runtime is reduced thanks to warm start initialization. Having estimated a Markov model from data, we employ it to optimize a tile-based 360 video streaming system. Extensive experiments show that our head movement prediction scheme noticeably outperformed existing proposals, and our optimized tile-based streaming scheme outperformed competitors in rate-distortion performance.
登录
查看更多内容
影响因子：
1.8
作者：
N. C. Nilsson;Tabitha C. Peck;G. Bruder;Eric Hodgson;S. Serafin;M. Whitton;Frank Steinicke;Evan Suma Rosenberg
通讯作者：
N. C. Nilsson;Tabitha C. Peck;G. Bruder;Eric Hodgson;S. Serafin;M. Whitton;Frank Steinicke;Evan Suma Rosenberg
DOI：
10.1145/3241539.3241565
发表时间：
2018-10
期刊：
Proceedings of the 24th Annual International Conference on Mobile Computing and Networking
影响因子：
--
作者：
Feng Qian;B. Han;Q. Xiao;V. Gopalakrishnan
通讯作者：
Feng Qian;B. Han;Q. Xiao;V. Gopalakrishnan
DOI：
10.1109/icassp40776.2020.9053598
发表时间：
2020-05
期刊：
ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
影响因子：
--
作者：
Xue Zhang;Gene Cheung;P. Callet;Jack Z. G. Tan
通讯作者：
Xue Zhang;Gene Cheung;P. Callet;Jack Z. G. Tan
DOI：
10.1145/3123266.3123339
发表时间：
2017-10
期刊：
Proceedings of the 25th ACM international conference on Multimedia
影响因子：
--
作者：
Mengbai Xiao;Chao Zhou;Yao Liu;Songqing Chen
通讯作者：
Mengbai Xiao;Chao Zhou;Yao Liu;Songqing Chen
DOI：
10.1145/2910017.2910641
发表时间：
2016-05
期刊：
Proceedings of the 7th International Conference on Multimedia Systems
影响因子：
--
作者：
J. L. Feuvre;C. Concolato
通讯作者：
J. L. Feuvre;C. Concolato