Domain Transfer with Generative Models and Neural Rendering

使用生成模型和神经渲染进行域转移

• 批准号: 453990920
• 负责人: Professor Dr.-Ing. Matthias Nießner
• 金额: --
• 依托单位: Lehrstuhl für Informatik XV: Graphik und Visualisierung
• 依托单位国家: 德国
• 项目类别: Research Units
• 资助国家: 德国
• 项目状态: 未结题
• 来源: https://gepris.dfg.de/gepris/projekt/453990920?language=en
• 关键词: Domain; Transfer; Generative; Models; Neural

In the recent years, we have seen the tremendous success of neural networks in almost every field of computer science. Nonetheless, despite this success, a fundamental limitation remains: the availability of labeled training data, which in general is costly and difficult to obtain, in particular for computer image tasks such as semantic segmentation where class labels must be manually provided for each pixel. A potential approach to tackling this problem is to exploit synthetic imagery as training data for neural networks; here, ground truth labels are already provided for free, and an virtually arbitrarily large amount of imagery from different viewpoints can be synthesized from a given 3D scene description. This potential has already inspired computer vision research to develop simulation environments in order to provide generate training data from these representations; e.g., Habitat and Gibson.The overarching goal of this proposal is to leverage training data across domains by bridging the domain gap between simulated and real-world visual data. Early works have proposed domain adaption techniques to address this challenging problems, such as the popular open set domain adaption method; however, the problem itself still remains challenging due to the mismatch in the underlying data statistics. In order to address the problem, we propose to develop new generative models that enable domain transfer by learning to match the respective underlying data distributions in both source (simulated) and target (real world) domains. We believe that this is a very timely direction with respect to the developments in the research community, since we have now seen very promising work on generative neural networks for visual data. In particular, generative adversarial networks (GANs) can now produce photo-realistic imagery from a random distributions such shown by progressive GAN, BigGAN, and the very recent StyleGAN methods. However, also probabilistic auto-regressive models have made tremendous progress in the recent years with works ranging from the early PixelCNN to the very recent high-quality results such as VQ-VAE-2. With these new advances, we see a compelling opportunity to develop such techniques towards bridging the synthetic-real domain gap; that is, leveraging generative approaches to transform synthetic data to its photo-realistic counterpart.Our main insight is to leverage graphics-based 3D understanding of imagery in order to inform generative neural networks to address the domain gap. By learning explicit 3D parameterizations of scenes captured in images, we can take advantage of physically-based modeling of imaging and 3D spatial consistency, which a network would then need not learn but could focus on bridging the domain-specific characteristics of synthetic and real data.

近年来，我们看到神经网络在计算机科学的几乎每个领域都取得了巨大的成功。尽管如此，尽管取得了成功，但仍然存在一个基本的限制：标记的训练数据的可用性，这通常是昂贵且难以获得的，特别是对于计算机图像任务，例如语义分割，其中必须为每个像素手动提供类别标签。解决这个问题的一种潜在方法是利用合成图像作为神经网络的训练数据;这里，地面真实标签已经免费提供，并且可以从给定的3D场景描述中合成来自不同视点的几乎任意数量的图像。这种潜力已经激发了计算机视觉研究开发模拟环境，以便从这些表示中生成训练数据;例如，Habitat和吉布森。该提案的总体目标是通过弥合模拟和真实世界视觉数据之间的领域差距来利用跨领域的训练数据。 早期的工作已经提出了域自适应技术来解决这个具有挑战性的问题，如流行的开集域自适应方法;然而，由于底层数据统计的不匹配，问题本身仍然具有挑战性。为了解决这个问题，我们建议开发新的生成模型，通过学习来匹配源（模拟）和目标（真实的世界）域中各自的底层数据分布，从而实现域转移。我们相信，这是研究界发展的一个非常及时的方向，因为我们现在已经看到了用于视觉数据的生成神经网络的非常有前途的工作。特别是，生成对抗网络（GAN）现在可以从渐进GAN、BigGAN和最新的StyleGAN方法所示的随机分布中生成逼真的图像。然而，近年来，概率自回归模型也取得了巨大进展，其工作范围从早期的PixelCNN到最近的高质量结果（例如VQ-VAE-2）。随着这些新的进展，我们看到了一个引人注目的机会，开发这样的技术来弥合合成-真实领域的差距;也就是说，利用生成方法将合成数据转换为照片般逼真的对应物。我们的主要见解是利用基于图形的图像3D理解，以告知生成神经网络来解决领域差距。通过学习图像中捕获的场景的显式3D参数化，我们可以利用基于物理的成像建模和3D空间一致性，然后网络不需要学习，而是可以专注于桥接合成和真实的数据的特定领域特征。