Deep style transfer for 3D meshes

点击翻译按钮获取中文摘要

Video games, especially large-scale productions such as AAA games, contain thousands of 3D assets that have been crafted by tens or even hundreds of highly skilled artists. In order to give the game a desired artistic look, these artists first model the assets in 3D and assign them default texture. Then, they proceed to carefully "paint" additional stylistic details to the texture in order to give them a desired look. This process, commonly known as "inking", is tedious, time-consuming and must sometimes be repeated several times when the artistic direction changes. In this grant application, we propose to develop computer vision algorithms for automatically applying a desired style onto the base texture map of 3D objects, and as such reproduce the inking process done by artists. The algorithms must be automatic, should be able to learn the stylization automatically from examples, and should be generic enough to be applied to thousands of different 3D models. Our key idea is to rely on recent image-to-image translation approaches based on deep generative adversarial networks (GANs). To attain these objectives, we will first build a dataset of 3D objects and their stylized texture maps based on the database acquired by the industrial partner over the years. Then, we will experiemnt with image-to-image translation GANs applied to texture maps, and to 3D renders of the object. Finally, we will compare both approaches, and explore future directions to reduce the need for large training sets. This application will contribute to the video game industry in Canada, which contributed $3.7B in GDP in 2017. The partnership between U. Laval and Gearbox Studio Québec will both exploit the strengths in deep learning and computer vision of the research team at U. Laval and the expertise in video game development at Gearbox. In particular, the industrial partner possesses a unique dataset of thousands of stylized 3D models used in past game productions which will be leveraged in this work.