Learning-Based Wavelet Video Coding Using Deep Adaptive Lifting

人工智能产生的基于学习的方法已经成功地应用于图像和视频处理的各个领域。在有损图像压缩领域，与经典图像编码器相比，在率失真性能方面也取得了显着进展。该比率描述了对于某一再现保真度的最大可实现压缩。此外，经典的图像和视频编码器是基于可变速率的概念。这允许根据期望的重构质量提供各种比特率。可以通过向具有不同信道容量的网络提供相同编解码器来描述示例性用例。目前端到端的基于学习的训练方法的特点是具有良好的信号自适应性，与经典方法相比，压缩性能得到了改善。然而，一个关键的缺点是缺乏对神经网络功能方式的理解，这是由于深度学习架构通常不是系统设计的，而是以试错的方式手动设计的。此外，神经网络的训练需要很大的计算复杂度，因为可变速率通常通过分别训练多个模型来获得。因此，本研究提出一种新的可变速率学习为基础的视频编码器，将开发使用运动补偿小波提升。除了速率自适应性，空间和时间的可伸缩性实现，从而在一个完全可伸缩的比特流。该方法基于所谓的提升结构，提供了应用任何非线性操作而不损害变换的重构特性的优点。这也使得在提升结构内实现神经网络成为可能，从而提高小波提升的效率。学习的小波系数有望实现更好的信号自适应性和数据压缩。与端到端训练方法相比，所提出的方法的另一个优点是由于提升结构的众所周知的架构而更好地理解神经网络的功能方式。因此，可以直接跟踪和解释网络架构中的变化。此外，提升结构的特征在于提供完全就地计算，其不需要任何辅助存储器。将这种深度自适应提升结构应用于视频压缩到目前为止还没有被考虑过，并且描述了基于学习的视频压缩的一个有前途的新概念，将可变速率和高可解释性结合在一个模型中。

Learning-based methods resulting from artificial intelligence have been used successfully in various fields of image and video processing. In the field of lossy image compression significant progress regarding the rate-distortion performance compared to classic image coders has been achieved as well. This ratio describes the maximum achievable compression for a certain reproduction fidelity. Moreover, classic image and video coders are based on the concept of variable rates. This allows for providing various bit rates in dependence of the desired reconstruction quality. An exemplary use case can be described by supplying networks with varying channel capacities with the same codec. Current end-to-end trained learning-based methods are characterized by their good signal adaptivity, resulting in an improved compression performance compared to classic approaches. However, a crucial disadvantage is given by the lack of understanding regarding the manner of functioning of neural networks, which is caused by the fact that deep learning architectures are usually not designed systematically but manually in a trial-and-error fashion. Moreover, the training of neural networks requires large computational complexity, since variable rates are usually obtained by training multiple models separately. Therefore, in this research proposal a novel variable rate learning-based video coder shall be developed using motion compensated wavelet lifting. Besides rate adaptivity, spatial and temporal scalability is achieved, resulting in a fully scalable bit stream. This method is based on the so-called lifting structure offering the advantage of applying any non-linear operation without harming the reconstruction property of the transform. This also enables the possibility of implementing neural networks within the lifting structure and, thereby, increases the efficiency of the wavelet lifting. Learned wavelet coefficients are expected to achieve a better signal adaptivity and data compaction. In contrast to end-to-end trained methods, a further advantage of the proposed approach is given by the better understanding regarding the manner of functioning of neural networks due to the well-known architecture of the lifting structure. Thereby, changes in the network architectures can directly be tracked and interpreted. Moreover, the lifting structure is characterized by providing a fully in-place calculation, which does not need any auxiliary memory. Applying such a deep adaptive lifting structure to video compression has not been considered so far and describes a promising new concept for learning-based video compression, combining variable rates and high interpretability in one model.