Low-latency FoV-adaptive Coding and Streaming for Interactive 360° Video Streaming
Low-latency FoV-adaptive Coding and Streaming for Interactive 360° Video Streaming
复制标题
DOI：
10.1145/3394171.3413751
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
Yixiang Mao;Liyang Sun;Yong Liu;Yao Wang
中科院分区：
文献类型：
--
作者：
Yixiang Mao;Liyang Sun;Yong Liu;Yao Wang
虚拟现实（VR）和增强现实（AR）技术近年来变得流行。编码和传输全向或360 ^\circ $视频是关键和具有挑战性的这些应用。360美元的视频比传统的平面视频需要更高的带宽。每秒120帧（fps）和24 K分辨率的高质量360美元视频可以轻松地消耗每秒1000比特的带宽。另一方面，在任何给定时间，用户仅观看其视场（FoV）内的$360^\circ$范围的一小部分。降低360 °视频带宽要求的一种有效方法是通过FoV自适应流，其以较高质量编码和传送预测的FoV区域，并丢弃或以较低质量编码剩余区域。这种策略已经在视频点播\citefov_adapt_2，fov_adapt_3，1，tile_based_3，qian 2016优化和实时视频流应用\citelive_1，live_2，live_3，sun 2020 flocking中得到了广泛的研究。交互式应用程序，如会议，游戏和远程协作，也可以受益于360^\circ $视频，通过创建一个身临其境的环境，让参与者相互互动。然而，交互式应用所需的具有极低延迟的360美元视频的实时编码和流式传输尚未得到充分解决。这项工作的重点是开发低延迟和FoV自适应编码和流媒体策略的交互式360 ^\circ $视频流。我们假设发送方和接收方通过具有动态变化的吞吐量的网络路径连接，而没有短延迟保证。发送方是视频源或中继源视频的代理服务器。接收方可以是直接呈现视频的终端用户设备，也可以是呈现视频并传输给终端用户的本地边缘服务器\citeHou2017。
Virtual Reality (VR) and Augmented Reality (AR) technologies have become popular in recent years. Encoding and transmitting the omni-directional or $360^\circ $ video is critical and challenging for those applications. The $360^\circ $ video requires much higher bandwidth than the traditional planar video. A premium quality $360^\circ $ video with 120 frames per second (fps) and 24K resolution can easily consume bandwidth in the range of Gigabits-per-second~\cite1. On the other hand, at any given time, a user only watches a small portion of the $360^\circ$ scope within her Field-of-View (FoV). An effective way to reduce the bandwidth requirement of $360^\circ $ video is through FoV-adaptive streaming, which codes and delivers the predicted FoV region at higher quality, and discards or codes at lower quality the remaining regions. Such strategy has been quite extensively studied for video-on-demand \citefov_adapt_2,fov_adapt_3,1,tile_based_3,qian2016optimizing and live video streaming applications\citelive_1,live_2,live_3, sun2020flocking. Interactive applications, such as conferencing, gaming, and remote collaboration, can also benefit from $360^\circ $ video by creating an immersive environment for participants to interact with each other citeinteractive_gamming \citevr_conferencing \citelee2015outatime. However, realtime coding and streaming of $360^\circ $ video with extremely low latency, required for interactive applications, has not been sufficiently addressed. This work focuses on developing low-latency and FoV-adaptive coding and streaming strategies for interactive $360^\circ$ video streaming. We assume the sender and the receiver are connected by a network path with dynamically varying throughput without short-latency guarantee. The sender is either the video source, or a proxy server relaying the source video. The receiver is either the end user device that directly renders the video, or a local edge server that renders the video and transmit to the end user \citeHou2017.