Extending the Frontiers of Deep Generative Modelling.

现代基于神经网络的模型在合成图像的质量方面取得了巨大的进步。然而，这些生成模型合成视频或三维（3D）场景的能力却很少受到研究者的关注。在本提案中，我描述了一系列三个项目，目标是解决这一差距：学习3D场景的生成模型；视频学习生成模型；学习不同表示之间的转换，特别是2D图像和3D场景之间的转换。在开发良好的3D场景表示时，3D表示是关键。在这里，我们建议使用一种称为splats的表示法。Splats将3D场景表示为一组独立的表面点。我们的方法将是建立一个生成模型，该模型将隐式地表示完整的3D场景，并仅输出与观察者当前视点对应的splats。这种方法应该允许我们以一种尚未被证明的方式扩展到具有杂乱物体的复杂场景。我们的视频方法将集中在两条创新轨道上。我们将探索新颖的模型架构（即不同模块之间的布线模式），以更好地捕捉视频的本质。具体来说，我们将探索允许像素从一帧传播到下一帧相对不变的简单途径的模型。这捕捉到了从一帧到另一帧，视频变化缓慢的观察结果。另一个创新的轨道包括探索学习算法的替代目标。我们将实验一种新的训练策略的变体，该策略将生成模型的学习框架为生成器和鉴别器之间的对抗游戏，鉴别器试图区分真实数据示例和来自生成器的合成示例。最后，我们将探索学习2D图像和3D场景之间转换的方法。为了做到这一点，我们将使用如上所述的相同的对抗性训练策略。我们在这个项目中面临的一个重要挑战是，从图像到3D场景的转换没有完全指定，例如，遮挡物体的表面没有在图像中表示，但它们必须在3D表示中推断出来。处理信息缺失问题是这部分建议的重点。有了这个提议，我们开始降低我们生活的动态3D世界与视频游戏和模拟器的虚拟世界之间的障碍。以这种方式提供丰富模拟的能力，有可能大大提高机器人和自主设计等领域的进步速度。

Modern neural network-based models have made dramatic improvement in the quality of synthesize images. However the ability of these generative models to synthesize video or 3 dimensional (3D) scenes has received much less attention from researchers. In this proposal, I describe a series three of projects with the goal of addressing this gap: learning generative models of 3D scenes; learning generative models of video; and learning transformations between different representations, specifically between 2D images and 3D scenes.In developing good representations of 3D scenes, the 3D representation is key. Here we propose to use a representation known as splats. Splats represent 3D scenes as a set of independent surface points. Our approach will be to build a generative model that will implicitly represent the full 3D scene and output only the splats that correspond to the current viewpoint of the observer. This approach should allow us to scale to complex scenes with cluttered objects in a way that has not yet been demonstrated.Our approach to video will focus on two tracks of innovations. We will explore novel model architectures (i.e. the wiring pattern between different modules) that better capture the nature of video. Specifically, we will explore models that allow an easy pathway for pixel from one frame to propagate to the next frame relatively unchanged. This captures the observation that from frame to frame, videos change slowly. The other track of innovation includes exploring alternative objective for the learning algorithm. We will experiment with a variant of a new training strategy that frames the learning of a generative model as an adversarial game between the generator and a discriminator that is trying to distinguish between true data examples and the synthesized examples from the generator.Finally, we will explore methods to learn transformations between 2D images and 3D scenes. To do so, we will use the same adversarial training strategy as described above. One important challenge that we face in this project is that the transformation from images to 3D scenes is not fully specified, for instance the surfaces of occluded objects are not represented in the image, yet they would have to be inferred in the 3D representation. Dealing with this issue of missing information is the focus of this part of the proposal.With this proposal, we set out to lower the barrier between the dynamic 3D world in which we live and the virtual world of video gams and simulators. Providing the ability to enriching simulation this way has the potential to greatly increase the rate of progress in fields such as robotics and autonomous design.