Extending the Frontiers of Deep Generative Modelling.

扩展深度生成建模的前沿。

• 批准号: RGPIN-2018-05920
• 负责人: Courville, Aaron
• 金额: $ 10.78万
• 依托单位: Université de Montréal
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2022
• 资助国家: 加拿大
• 起止时间: 2022-01-01 至 2023-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=757415
• 关键词: Extending; Frontiers; Deep; Generative; Modelling.

Modern neural network-based models have made dramatic improvement in the quality of synthesize images. However the ability of these generative models to synthesize video or 3 dimensional (3D) scenes has received much less attention from researchers. In this proposal, I describe a series three of projects with the goal of addressing this gap: learning generative models of 3D scenes; learning generative models of video; and learning transformations between different representations, specifically between 2D images and 3D scenes.In developing good representations of 3D scenes, the 3D representation is key. Here we propose to use a representation known as splats. Splats represent 3D scenes as a set of independent surface points. Our approach will be to build a generative model that will implicitly represent the full 3D scene and output only the splats that correspond to the current viewpoint of the observer. This approach should allow us to scale to complex scenes with cluttered objects in a way that has not yet been demonstrated.Our approach to video will focus on two tracks of innovations. We will explore novel model architectures (i.e. the wiring pattern between different modules) that better capture the nature of video. Specifically, we will explore models that allow an easy pathway for pixel from one frame to propagate to the next frame relatively unchanged. This captures the observation that from frame to frame, videos change slowly. The other track of innovation includes exploring alternative objective for the learning algorithm. We will experiment with a variant of a new training strategy that frames the learning of a generative model as an adversarial game between the generator and a discriminator that is trying to distinguish between true data examples and the synthesized examples from the generator.Finally, we will explore methods to learn transformations between 2D images and 3D scenes. To do so, we will use the same adversarial training strategy as described above. One important challenge that we face in this project is that the transformation from images to 3D scenes is not fully specified, for instance the surfaces of occluded objects are not represented in the image, yet they would have to be inferred in the 3D representation. Dealing with this issue of missing information is the focus of this part of the proposal.With this proposal, we set out to lower the barrier between the dynamic 3D world in which we live and the virtual world of video gams and simulators. Providing the ability to enriching simulation this way has the potential to greatly increase the rate of progress in fields such as robotics and autonomous design.

现代神经网络模型在合成图像质量方面取得了巨大的进步。然而，这些生成模型合成视频或三维（3D）场景的能力受到研究人员的关注要少得多。在这个建议中，我描述了一系列的三个项目，旨在解决这个差距：学习3D场景的生成模型;学习视频的生成模型;学习不同表示之间的转换，特别是2D图像和3D场景之间的转换。在开发3D场景的良好表示时，3D表示是关键。在这里，我们建议使用称为splats的表示。Splats将3D场景表示为一组独立的曲面点。我们的方法将是构建一个生成模型，该模型将隐式地表示完整的3D场景，并仅输出与观察者的当前视点相对应的splats。这种方法应该允许我们以一种尚未被证明的方式扩展到对象杂乱的复杂场景。我们的视频方法将集中在两个创新轨道上。我们将探索新的模型架构（即不同模块之间的布线模式），以更好地捕捉视频的本质。具体来说，我们将探索允许像素从一帧传播到下一帧相对不变的简单路径的模型。这捕获了从帧到帧的观察，视频变化缓慢。创新的另一个轨道包括探索学习算法的替代目标。我们将尝试一种新的训练策略的变体，该策略将生成模型的学习框架为生成器和生成器之间的对抗游戏，该生成器试图区分真实数据示例和生成器合成的示例。最后，我们将探索学习2D图像和3D场景之间转换的方法。为此，我们将使用与上述相同的对抗训练策略。我们在这个项目中面临的一个重要挑战是，从图像到3D场景的转换没有完全指定，例如，被遮挡物体的表面没有在图像中表示，但它们必须在3D表示中推断出来。解决信息缺失的问题是这部分提案的重点。通过这一提案，我们开始降低我们生活的动态3D世界与视频游戏和模拟器的虚拟世界之间的障碍。以这种方式提供丰富模拟的能力有可能大大提高机器人和自主设计等领域的进步速度。