FOCAS: Practical Video Super Resolution using Foveated Rendering
FOCAS: Practical Video Super Resolution using Foveated Rendering
复制标题
DOI：
10.1145/3474085.3475673
复制
发表时间：
2021-10
期刊：
影响因子：
--
通讯作者：
Lingdong Wang;M. Hajiesmaili;R. Sitaraman
中科院分区：
文献类型：
--
作者：
Lingdong Wang;M. Hajiesmaili;R. Sitaraman
超分辨率（SR）是一种从低分辨率（LR）图像重建高分辨率（HR）图像的技术。SR对于视频流具有很大的希望，因为LR视频片段可以从视频服务器传输到客户端，然后客户端使用SR重建HR版本，从而显著减少网络带宽。然而，SR很少用于实际的实时视频流，因为帧重建的计算开销导致大的延迟和低的帧速率。为了减少计算开销并使SR实用，我们提出了一种基于深度学习的SR方法，称为Fo veated Cas caded Video Super Resolution（focas）。FOCAS依赖于人眼仅在视网膜的微小中央凹区域中具有高敏锐度的事实。FOCAS在中央凹区域中使用更多的神经网络块，以提供更高的视频质量，而在外围使用更少的块，因为较低的质量就足够了。为了优化计算资源并减少重建延迟，focas制定并解决了一个凸优化问题，以决定在帧的每个区域中使用的神经网络块的数量。使用广泛的实验，我们表明，focas减少了50%-70%的延迟，同时保持可比的视觉质量作为传统的（非中心凹）SR。此外，focas提供了一个12- 16倍的减少客户端到服务器的网络带宽相比，发送完整的HR视频片段。
Super-resolution (SR) is a well-studied technique for reconstructing high-resolution (HR) images from low-resolution (LR) ones. SR holds great promise for video streaming since an LR video segment can be transmitted from the video server to the client that then reconstructs the HR version using SR, resulting in a significant reduction in network bandwidth. However, SR is seldom used in practice for real-time video streaming, because the computational overhead of frame reconstruction results in large latency and low frame rate. To reduce the computational overhead and make SR practical, we propose a deep-learning-based SR method called Fo veated Cas caded Video Super Resolution (focas). focas relies on the fact that human eyes only have high acuity in a tiny central foveal region of the retina. focas uses more neural network blocks in the foveal region to provide higher video quality, while using fewer blocks in the periphery as lower quality is sufficient. To optimize the computational resources and reduce reconstruction latency, focas formulates and solves a convex optimization problem to decide the number of neural network blocks to use in each region of the frame. Using extensive experiments, we show that focas reduces the latency by 50%-70% while maintaining comparable visual quality as traditional (non-foveated) SR. Further, focas provides a 12-16x reduction in the client-to-server network bandwidth in comparison with sending the full HR video segments.