VertexShuffle-Based Spherical Super-Resolution for 360-Degree Videos
VertexShuffle-Based Spherical Super-Resolution for 360-Degree Videos
复制标题
DOI：
10.1145/3649315
复制
发表时间：
2024-02
期刊：
影响因子：
--
通讯作者：
Na Li;Yao Liu
中科院分区：
文献类型：
--
作者：
Na Li;Yao Liu
360-度视频是一种新兴的媒体形式，其对关于相机周围的所有方向的信息进行编码，从而向用户提供沉浸式体验。与传统的2D视频不同，360度视频中的视觉信息可以自然地表示为球体上的像素。受最先进的基于深度学习的2D图像超分辨率模型和球形CNN的启发，本文设计了一种用于360度视频的新型球形超分辨率（SSR）方法。为了支持360度视频数据的视口自适应和带宽高效传输/流式传输并节省计算，我们提出了聚焦二十面体网格来表示球体上的一个小区域。我们进一步构造矩阵，将整个球体上的球形内容旋转到聚焦网格区域，使我们能够使用聚焦网格来表示球体上的任何区域。受PixelShuffle二维超分辨率操作的启发，我们还提出了一种新的网格上的VertexShuffle操作和改进版本VertexShuffle_V2。我们将我们的SSR方法与最先进的2D超分辨率模型进行了比较，并表明SSR在应用于球形信号时有可能获得显着的好处。
360-degree video is an emerging form of media that encodes information about all directions surrounding a camera, offering an immersive experience to the users. Unlike traditional 2D videos, visual information in 360-degree videos can be naturally represented as pixels on a sphere. Inspired by state-of-the-art deep-learning-based 2D image super-resolution models and spherical CNNs, in this paper, we design a novel spherical super-resolution (SSR) approach for 360-degree videos. To support viewport-adaptive and bandwidth-efficient transmission/streaming of 360-degree video data and save computation, we propose the Focused Icosahedral Mesh to represent a small area on the sphere. We further construct matrices to rotate spherical content over the entire sphere to the focused mesh area, allowing us to use the focused mesh to represent any area on the sphere. Motivated by the PixelShuffle operation for 2D super-resolution, we also propose a novel VertexShuffle operation on the mesh and an improved version VertexShuffle_V2. We compare our SSR approach with state-of-the-art 2D super-resolution models and show that SSR has the potential to achieve significant benefits when applied to spherical signals.