Extending the Frontiers of Deep Generative Modelling.

扩展深度生成建模的前沿。

• 批准号: RGPIN-2018-05920
• 负责人: Courville, Aaron
• 金额: $ 5.39万
• 依托单位: Université de Montréal
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2020
• 资助国家: 加拿大
• 起止时间: 2020-01-01 至 2021-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=713947
• 关键词: Extending; Frontiers; Deep; Generative; Modelling.

Modern neural network-based models have made dramatic improvement in the quality of synthesize images. However the ability of these generative models to synthesize video or 3 dimensional (3D) scenes has received much less attention from researchers. In this proposal, I describe a series three of projects with the goal of addressing this gap: learning generative models of 3D scenes; learning generative models of video; and learning transformations between different representations, specifically between 2D images and 3D scenes. In developing good representations of 3D scenes, the 3D representation is key. Here we propose to use a representation known as splats. Splats represent 3D scenes as a set of independent surface points. Our approach will be to build a generative model that will implicitly represent the full 3D scene and output only the splats that correspond to the current viewpoint of the observer. This approach should allow us to scale to complex scenes with cluttered objects in a way that has not yet been demonstrated. Our approach to video will focus on two tracks of innovations. We will explore novel model architectures (i.e. the wiring pattern between different modules) that better capture the nature of video. Specifically, we will explore models that allow an easy pathway for pixel from one frame to propagate to the next frame relatively unchanged. This captures the observation that from frame to frame, videos change slowly. The other track of innovation includes exploring alternative objective for the learning algorithm. We will experiment with a variant of a new training strategy that frames the learning of a generative model as an adversarial game between the generator and a discriminator that is trying to distinguish between true data examples and the synthesized examples from the generator. Finally, we will explore methods to learn transformations between 2D images and 3D scenes. To do so, we will use the same adversarial training strategy as described above. One important challenge that we face in this project is that the transformation from images to 3D scenes is not fully specified, for instance the surfaces of occluded objects are not represented in the image, yet they would have to be inferred in the 3D representation. Dealing with this issue of missing information is the focus of this part of the proposal. With this proposal, we set out to lower the barrier between the dynamic 3D world in which we live and the virtual world of video gams and simulators. Providing the ability to enriching simulation this way has the potential to greatly increase the rate of progress in fields such as robotics and autonomous design.

现代基于神经网络的模型在合成图像的质量方面取得了巨大的进步。然而，这些生成模型合成视频或三维（3D）场景的能力却很少受到研究者的关注。在本提案中，我描述了一系列三个项目，目标是解决这一差距：学习3D场景的生成模型；视频学习生成模型；学习不同表示之间的转换，特别是2D图像和3D场景之间的转换。