Domain Transfer with Generative Models and Neural Rendering

近年来，我们看到神经网络在计算机科学的几乎每一个领域都取得了巨大的成功。然而，尽管取得了这样的成功，但仍然存在一个基本的限制：标签训练数据的可用性，这通常是昂贵和难以获得的，特别是对于诸如语义分割之类的计算机图像任务，其中必须为每个像素手动提供类别标签。解决这个问题的一个潜在方法是利用合成图像作为神经网络的训练数据；在这里，已经免费提供了地面真实标签，并且可以从给定的3D场景描述合成来自不同视点的几乎任意数量的图像。这种潜力已经激发了计算机视觉研究，以开发模拟环境，以便从这些表示提供生成的训练数据，例如，人居和吉布森。该提议的首要目标是通过弥合模拟和真实世界视觉数据之间的领域差距来利用跨域的训练数据。早期的工作已经提出了领域自适应技术来解决这一挑战性问题，例如流行的开集领域自适应方法；然而，由于底层数据统计的不匹配，该问题本身仍然具有挑战性。为了解决这个问题，我们建议开发新的生成性模型，通过学习匹配源(模拟)和目标(现实世界)域中各自的底层数据分布来实现域转移。我们认为，就研究界的发展而言，这是一个非常及时的方向，因为我们现在已经看到了关于视觉数据生成神经网络的非常有前途的工作。特别是，生成性对抗网络(GAN)现在可以从随机分布中产生照片级的图像，如渐进式GAN、BigGAN和最新的StyleGAN方法。然而，概率自回归模型在最近几年也取得了巨大的进步，从早期的PixelCNN到最近的高质量结果，如VQ-VAE-2。有了这些新的进展，我们看到了一个令人信服的机会来开发这样的技术来弥合合成和真实领域的差距；即利用生成性方法将合成数据转换为照片级的对应数据。我们的主要见解是利用基于图形的3D图像理解，以便为生成性神经网络提供信息来解决领域差距。通过学习图像中捕获的场景的显式3D参数，我们可以利用基于物理的成像建模和3D空间一致性，这样网络就不需要学习，而是可以专注于连接合成数据和真实数据的特定领域特征。

In the recent years, we have seen the tremendous success of neural networks in almost every field of computer science. Nonetheless, despite this success, a fundamental limitation remains: the availability of labeled training data, which in general is costly and difficult to obtain, in particular for computer image tasks such as semantic segmentation where class labels must be manually provided for each pixel. A potential approach to tackling this problem is to exploit synthetic imagery as training data for neural networks; here, ground truth labels are already provided for free, and an virtually arbitrarily large amount of imagery from different viewpoints can be synthesized from a given 3D scene description. This potential has already inspired computer vision research to develop simulation environments in order to provide generate training data from these representations; e.g., Habitat and Gibson.The overarching goal of this proposal is to leverage training data across domains by bridging the domain gap between simulated and real-world visual data. Early works have proposed domain adaption techniques to address this challenging problems, such as the popular open set domain adaption method; however, the problem itself still remains challenging due to the mismatch in the underlying data statistics. In order to address the problem, we propose to develop new generative models that enable domain transfer by learning to match the respective underlying data distributions in both source (simulated) and target (real world) domains. We believe that this is a very timely direction with respect to the developments in the research community, since we have now seen very promising work on generative neural networks for visual data. In particular, generative adversarial networks (GANs) can now produce photo-realistic imagery from a random distributions such shown by progressive GAN, BigGAN, and the very recent StyleGAN methods. However, also probabilistic auto-regressive models have made tremendous progress in the recent years with works ranging from the early PixelCNN to the very recent high-quality results such as VQ-VAE-2. With these new advances, we see a compelling opportunity to develop such techniques towards bridging the synthetic-real domain gap; that is, leveraging generative approaches to transform synthetic data to its photo-realistic counterpart.Our main insight is to leverage graphics-based 3D understanding of imagery in order to inform generative neural networks to address the domain gap. By learning explicit 3D parameterizations of scenes captured in images, we can take advantage of physically-based modeling of imaging and 3D spatial consistency, which a network would then need not learn but could focus on bridging the domain-specific characteristics of synthetic and real data.