Extending the Frontiers of Deep Generative Modelling.

点击翻译按钮获取中文摘要

Modern neural network-based models have made dramatic improvement in the quality of synthesize images. However the ability of these generative models to synthesize video or 3 dimensional (3D) scenes has received much less attention from researchers. In this proposal, I describe a series three of projects with the goal of addressing this gap: learning generative models of 3D scenes; learning generative models of video; and learning transformations between different representations, specifically between 2D images and 3D scenes.******In developing good representations of 3D scenes, the 3D representation is key. Here we propose to use a representation known as splats. Splats represent 3D scenes as a set of independent surface points. Our approach will be to build a generative model that will implicitly represent the full 3D scene and output only the splats that correspond to the current viewpoint of the observer. This approach should allow us to scale to complex scenes with cluttered objects in a way that has not yet been demonstrated.******Our approach to video will focus on two tracks of innovations. We will explore novel model architectures (i.e. the wiring pattern between different modules) that better capture the nature of video. Specifically, we will explore models that allow an easy pathway for pixel from one frame to propagate to the next frame relatively unchanged. This captures the observation that from frame to frame, videos change slowly. The other track of innovation includes exploring alternative objective for the learning algorithm. We will experiment with a variant of a new training strategy that frames the learning of a generative model as an adversarial game between the generator and a discriminator that is trying to distinguish between true data examples and the synthesized examples from the generator.******Finally, we will explore methods to learn transformations between 2D images and 3D scenes. To do so, we will use the same adversarial training strategy as described above. One important challenge that we face in this project is that the transformation from images to 3D scenes is not fully specified, for instance the surfaces of occluded objects are not represented in the image, yet they would have to be inferred in the 3D representation. Dealing with this issue of missing information is the focus of this part of the proposal.******With this proposal, we set out to lower the barrier between the dynamic 3D world in which we live and the virtual world of video gams and simulators. Providing the ability to enriching simulation this way has the potential to greatly increase the rate of progress in fields such as robotics and autonomous design.