Generative models for 3D textures

点击翻译按钮获取中文摘要

3D textures are the volumetric equivalent of 2D raster textures. They offer several advantages over their 2D counterparts. However, despite recent progress and unlike 2D textures, dedicated techniques that allow to model and create such textures are limited. Unlike scanning a 2D image, digitization of volumetric color information is usually impractical. This creates the need for efficient and controllable synthesis algorithms. One reason is the lack of intuitive specification models allowing a user to indicate what he has in mind. Another reason is the lack of scalable models that can handle three dimensions at high resolution. Our main research objective is to pursue fundamental and applied research for the development of novel representations and design algorithms of 3D textures. We target four specific objectives. 1) User centered design models for controlling 3D textures synthesis. First, we will offer a highly controllable synthesis method. This raises a major issue, especially when tackling textures with distinguishable patterns. Unlike 2D synthesis, where the user can provide a small example to input the algorithm, providing 3D examples is impractical. We will propose novel controls using a conditionnal adversarial network. 2) Structural sound models for enriching 3D printing. 3D printing offers a challenging context allowing designers and artists with an exciting opportunity to imagine and produce complex patterns and objects. We will propose a method that tackles structural soundness for printed 3D textures based objects. 3) Temporal coherent models for video style transferring. This is the task of automatically combining the content of one video and the style of an image into a stylized output video. Typically, the style of an artistic image is transferred to a photo-realistic video. We propose to use a 3D convolutional neural network to stylize local video blocks, several frames at a time. At the texture scale, this will guarantee consistency in both spatial and temporal dimensions, avoiding the estimation of the optical flow. 4) Edge--aware models for data- augmentation in medical applications. Deep learning techniques have been successful in medical image analysis. However, large labeled training databases are needed to achieve good performance. Such data is usually difficult to gather since it usually needs manual labeling. We will evaluate how data augmentation through 3D constrained texture synthesis can improve performances. We will focus on modalities relying on texture features such as histology and MRI. The proposed research program will develop tools and techniques enabling breakthrough innovation, leading to major advances in several applications, related to the Canadian computer animation and video games industry, and the medical field.