Learning-Based Wavelet Video Coding Using Deep Adaptive Lifting

源于人工智能的基于学习的方法已经成功地应用于图像和视频处理的各个领域。在有损图像压缩领域，与经典图像编码器相比，在率失真性能方面也取得了显着进展。该比率描述了在一定的再现保真度下可实现的最大压缩。此外，传统的图像和视频编码器基于可变速率的概念。这允许根据期望的重建质量提供各种比特率。可以通过向具有不同信道容量的网络提供相同的编解码器来描述示例性用例。当前基于端到端训练的学习方法具有良好的信号自适应性，与经典方法相比，压缩性能得到了改善。然而，一个严重的缺点是缺乏对神经网络运行方式的理解，这是由于深度学习体系结构通常不是系统地设计，而是以试错的方式手动设计的。此外，神经网络的训练需要很大的计算复杂度，因为可变速率通常是通过分别训练多个模型来获得的。因此，在本研究方案中，将利用运动补偿小波提升技术开发一种新的基于可变速率学习的视频编码器。除了速率自适应，还实现了空间和时间可伸缩性，从而产生完全可伸缩的比特流。这种方法是基于所谓的提升结构，提供了在不损害变换的重构性能的情况下应用任何非线性运算的优点。这也使得在提升结构内实现神经网络成为可能，从而提高了小波提升的效率。学习后的小波系数有望达到更好的信号自适应性和数据紧凑性。与端到端训练的方法相比，由于提升结构的众所周知的结构，更好地理解了神经网络的运行方式，从而使所提出的方法具有更大的优势。因此，可以直接跟踪和解释网络体系结构中的变化。此外，该吊装结构的特点是提供完全原地计算，不需要任何辅助存储器。将这种深度自适应提升结构应用到视频压缩中，到目前为止还没有人考虑过，它描述了一种基于学习的视频压缩的新概念，在一个模型中结合了可变速率和高可解释性。

Learning-based methods resulting from artificial intelligence have been used successfully in various fields of image and video processing. In the field of lossy image compression significant progress regarding the rate-distortion performance compared to classic image coders has been achieved as well. This ratio describes the maximum achievable compression for a certain reproduction fidelity. Moreover, classic image and video coders are based on the concept of variable rates. This allows for providing various bit rates in dependence of the desired reconstruction quality. An exemplary use case can be described by supplying networks with varying channel capacities with the same codec. Current end-to-end trained learning-based methods are characterized by their good signal adaptivity, resulting in an improved compression performance compared to classic approaches. However, a crucial disadvantage is given by the lack of understanding regarding the manner of functioning of neural networks, which is caused by the fact that deep learning architectures are usually not designed systematically but manually in a trial-and-error fashion. Moreover, the training of neural networks requires large computational complexity, since variable rates are usually obtained by training multiple models separately. Therefore, in this research proposal a novel variable rate learning-based video coder shall be developed using motion compensated wavelet lifting. Besides rate adaptivity, spatial and temporal scalability is achieved, resulting in a fully scalable bit stream. This method is based on the so-called lifting structure offering the advantage of applying any non-linear operation without harming the reconstruction property of the transform. This also enables the possibility of implementing neural networks within the lifting structure and, thereby, increases the efficiency of the wavelet lifting. Learned wavelet coefficients are expected to achieve a better signal adaptivity and data compaction. In contrast to end-to-end trained methods, a further advantage of the proposed approach is given by the better understanding regarding the manner of functioning of neural networks due to the well-known architecture of the lifting structure. Thereby, changes in the network architectures can directly be tracked and interpreted. Moreover, the lifting structure is characterized by providing a fully in-place calculation, which does not need any auxiliary memory. Applying such a deep adaptive lifting structure to video compression has not been considered so far and describes a promising new concept for learning-based video compression, combining variable rates and high interpretability in one model.