Conditional Coding for Learned Image and Video Compression

汉诺威莱布尼茨大学(LUH)信息研究所(TNT)和台湾交通大学(NYCU)计算机科学系的这一联合研究项目从条件编码的角度研究了端到端的学习视频压缩，并采用了基于元学习的正则化和剪裁方案。深度学习的到来推动了端到端学习压缩的发展。最近几年见证了学习图像压缩的成功，最先进的MS-SSIM结果显示出比VVC Intra更好的结果(以及与VVC Intra相当的PSNR结果)。相比之下，端到端学习视频压缩的发展还处于早期阶段。大多数已学习的视频编解码器遵循传统的基于混合的编码体系结构，即先进行时间预测，然后进行基于变换的残差编码。最近的一份报告表明，虽然最新的学习视频编解码器比x265具有更好的效果，但在更真实的测试条件下，它们很难与HEVC测试模型(HM)竞争。最近，一种被称为帧间条件编码的新流派出现了，将端到端的学习视频编码的压缩性能提升到了一个新的水平。条件编码的思想是根据有用的上下文信息来学习编码帧的数据分布，以达到更低的条件熵率以获得更好的压缩。深度生成模型的出现，如变分自动编码器(VAE)和归一化流模型，为基于学习的压缩范式的转变开辟了新的机会。目前，VAE是压缩主干的流行选择。作为一种新的尝试，该联合研究方案引入了一种特殊类型的归一化流模型，称为扩展归一化流(ANF)，用于条件编码。我们选择ANF是因为ANF具有比VAE更好的表现力，并将VAE作为特例包括在内。这项联合研究项目的另一个值得注意的方面是解决学习的视频编解码器的泛化和适应性。学习的编解码器经常受到训练数据和测试数据之间的域差距的影响；也就是说，他们可能不能很好地在看不见的数据上进行概括。在更一般的意义上，它们很难实现对单个测试图像/视频的最佳压缩，每个测试图像/视频实际上都可以被认为是一个不同的域。为了提高普适性，该建议将以元学习的形式结合Noether定理，以学习鼓励解码的视频帧在时间维度上保持一定的潜在一致性的归纳偏差。我们还将使用这种学习的感应偏差来在推断时调整编码器和/或解码器，以适应个别视频。由于其无监督的性质，我们的方法具有不需要在比特流中发出任何附加信息的显著特征。

This joint research project between the Institut für Informationsverarbeitung (TNT) of the Leibniz Universität Hannover (LUH) and the Department of Computer Science of the National Chiao Tung University (NYCU) in Taiwan addresses end-to-end learned video compression from the perspective of conditional coding with an meta learning-based regularization and tailoring scheme.The arrival of deep learning spurs a new wave of developments in end-to-end learned compression. Recent years witnessed the success of learned image compression, with the state-of-the-art showing better MS-SSIM results than (and comparable PSNR results to) VVC Intra. By comparison, the development of end-to-end learned video compression is still in its early stage. Most learned video codecs follow the traditional, hybrid-based coding architecture, namely temporal prediction followed by transform-based residual coding. A recent publication indicates that although the state-of-the-art learned video codecs show better results than x265, they can hardly compete with the HEVC Test Model (HM) under more realistic test conditions.Recently, a new school of thought, known as inter-frame conditional coding, emerged, taking end-to-end learned video coding to a new level of compression performance. The idea of conditional coding is to learn the data distribution of a coding frame conditioned on useful contextual information, in order to reach a lower conditional entropy rate for better compression.The emergence of deep generative models, such as variational autoencoders (VAE) and normalizing flow models, opens up new opportunities for a paradigm shift in learning-based compression. Currently, VAE is a popular choice for the compression backbone. Representing a new attempt, this joint research proposal introduces a special type of normalizing flow model, called augmented normalizing flows (ANF), for conditional coding. We choose ANF because it is shown to achieve superior expressiveness to VAE and includes VAE as a special case.Another notable aspect of this joint research project is to address the generalizability and adaptability of the learned video codecs. The learned codecs often suffer from the domain gap between the training and the test data; that is, they may not generalize well on unseen data. In a more general sense, they can hardly achieve optimal compression for individual test images/videos, each of which can in fact be considered a distinct domain. To improve the generalizability, this proposal shall incorporate Noether’s theorem in the form of meta learning to learn an inductive bias that encourages decoded video frames to conserve certain latent consistency in the temporal dimension. We shall also use this learned inductive bias to adapt the encoder and/or the decoder at inference time to suit individual videos. Due to its unsupervised nature, our approach has the striking feature of not having to signal any additional information in the bitstream.