Projection-Based Ultra Wide-Angle and 360° Video Coding

监控和汽车应用都使用超广角摄像头来捕获超过180度的视野。使用360度摄像机，甚至可以实现360度的水平视野。这样的序列提供所捕获的场景的完整环绕视图，并且因此在虚拟现实应用的上下文中特别感兴趣，其中它们提供沉浸式体验。为了提供足够的视觉质量，使用4K或8K的单个视频帧的分辨率，这导致在没有压缩的情况下无法传输或存储的大量数据。传统的编码技术不考虑超广角和360度相机的非透视投影函数，导致次优编码增益。因此，本项目的目标是开发用于超广角和360度视频序列的新型压缩技术，以实现高效的存储和传输。在该项目的第一阶段中，已经开发了一种新颖的运动补偿技术，其比最先进的基于投影的方法和传统的平移方法平均高出2.40 dB和几乎3.00 dB，已经导出了对超过90度入射角的显著改进的处理，并建立了一个实时交互式可视化工具来演示所开发的算法。在项目的第二阶段，将在项目第一阶段的调查基础上，研究基于投影的方法如何以及在多大程度上提高混合视频编码和基于学习的视频压缩网络的压缩效率。因此，该项目致力于对混合视频编码的基于投影的新扩展的研究，其中特别努力探索考虑编码链中所有相关元素的整体方法。此外，由于有前途的端到端学习视频压缩技术的快速发展，基于投影的方法的适用性，端到端优化的视频压缩网络的研究，旨在提高超广角和360度视频的压缩效率。生成和发布合适的360度数据集，用于混合和基于学习的视频编码技术的培训和测试也是该项目的一部分。

Both surveillance and automotive applications make use of ultra wide-angle cameras to capture a field of view of more than 180 degrees. With 360-degree cameras, a horizontal field of view of 360 degrees is even possible. Such sequences offer a complete surround view of the captured scene and are thus of particular interest in the context of virtual reality applications, where they provide an immersive experience. In order to provide a sufficient visual quality, a resolution of the single video frames of 4K or 8K is used, which leads to an immense amount of data that cannot be transmitted or stored without compression. Conventional coding techniques do not consider the non-perspective projection functions of ultra wide-angle and 360-degree cameras, leading to suboptimal coding gains. The goal of this project is thus the development of novel compression techniques for ultra wide-angle and 360-degree video sequences to achieve an efficient storage and transmission. In a first phase of the project, a novel motion compensation technique has been developed that outperforms the the state-of-the-art projection-based approach and the traditional translational approach by 2.40 dB and almost 3.00 dB on average, a substantially improved handling of beyond 90-degrees incident angles has been derived, and a real-time interactive visualization tool demonstrating the developed algorithms has been established. In a second phase of the project, it shall be researched how and to what extend projection-based approaches can improve the compression efficiency of hybrid video coding and learning-based video compression network building upon the investigations from the first phase of the project. Thereby, the project addresses research on novel projection-based extensions to hybrid video coding, where special effort goes into the exploration of holistic approaches that take all dependent elements in the coding chain into account. Furthermore, due to the fast-paced development of promising end-to-end learned video compression techniques, the applicability of projection-based methods to end-to-end optimized video compression networks is researched aiming for improved compression efficiency of ultra wide-angle and 360-degree video. The generation and publication of suitable 360-degree datasets for training and testing of the developed techniques for hybrid and learning-based video coding is also part of the project.