Projection-Based Ultra Wide-Angle and 360° Video Coding

监控和汽车应用都使用超广角相机来捕捉超过180度的视野。使用360度相机，甚至可以实现360度的水平视野。这样的序列提供了捕获场景的完整环绕视图，因此在虚拟现实应用的背景下特别感兴趣，在那里它们提供了身临其境的体验。为了提供足够的视觉质量，使用4K或8K的单视频帧分辨率，这导致大量数据如果不压缩就无法传输或存储。传统的编码技术没有考虑到超广角和360度摄像机的非透视投影功能，导致编码效果不理想。因此，本项目的目标是为超广角和360度视频序列开发新的压缩技术，以实现高效的存储和传输。在项目的第一阶段，开发了一种新的运动补偿技术，该技术比最先进的基于投影的方法和传统的平移方法平均高出2.40 dB和近3.00 dB，大大改善了对超过90度入射角的处理，并建立了一个实时交互式可视化工具来展示所开发的算法。在项目的第二阶段，在项目第一期调研的基础上，研究基于投影的方法如何以及在多大程度上提高混合视频编码和基于学习的视频压缩网络的压缩效率。因此，该项目致力于研究基于新投影的混合视频编码扩展，其中特别努力探索考虑编码链中所有依赖元素的整体方法。此外，由于端到端学习视频压缩技术的快速发展，为了提高超广角和360度视频的压缩效率，研究了基于投影的方法在端到端优化视频压缩网络中的适用性。生成和出版合适的360度数据集，用于培训和测试已开发的混合和基于学习的视频编码技术，也是该项目的一部分。

Both surveillance and automotive applications make use of ultra wide-angle cameras to capture a field of view of more than 180 degrees. With 360-degree cameras, a horizontal field of view of 360 degrees is even possible. Such sequences offer a complete surround view of the captured scene and are thus of particular interest in the context of virtual reality applications, where they provide an immersive experience. In order to provide a sufficient visual quality, a resolution of the single video frames of 4K or 8K is used, which leads to an immense amount of data that cannot be transmitted or stored without compression. Conventional coding techniques do not consider the non-perspective projection functions of ultra wide-angle and 360-degree cameras, leading to suboptimal coding gains. The goal of this project is thus the development of novel compression techniques for ultra wide-angle and 360-degree video sequences to achieve an efficient storage and transmission. In a first phase of the project, a novel motion compensation technique has been developed that outperforms the the state-of-the-art projection-based approach and the traditional translational approach by 2.40 dB and almost 3.00 dB on average, a substantially improved handling of beyond 90-degrees incident angles has been derived, and a real-time interactive visualization tool demonstrating the developed algorithms has been established. In a second phase of the project, it shall be researched how and to what extend projection-based approaches can improve the compression efficiency of hybrid video coding and learning-based video compression network building upon the investigations from the first phase of the project. Thereby, the project addresses research on novel projection-based extensions to hybrid video coding, where special effort goes into the exploration of holistic approaches that take all dependent elements in the coding chain into account. Furthermore, due to the fast-paced development of promising end-to-end learned video compression techniques, the applicability of projection-based methods to end-to-end optimized video compression networks is researched aiming for improved compression efficiency of ultra wide-angle and 360-degree video. The generation and publication of suitable 360-degree datasets for training and testing of the developed techniques for hybrid and learning-based video coding is also part of the project.