Deep style transfer for 3D meshes

视频游戏，特别是大型制作，如AAA游戏，包含成千上万的3D资产，这些资产是由数十甚至数百名高技能的艺术家制作的。为了给游戏一个理想的艺术外观，这些艺术家首先在3D模型中的资产，并为它们分配默认的纹理。然后，他们继续仔细地“画”额外的风格细节的纹理，以给他们一个理想的外观。这个过程，通常被称为“上墨”，是乏味的，耗时的，有时必须重复几次时，艺术方向的变化。 在这项拨款申请中，我们建议开发计算机视觉算法，用于自动将所需的样式应用到3D对象的基本纹理图上，并因此再现艺术家完成的着墨过程。算法必须是自动的，应该能够从示例中自动学习样式化，并且应该足够通用，以应用于数千个不同的3D模型。我们的主要想法是依靠最近基于深度生成对抗网络（GAN）的图像到图像翻译方法。为了实现这些目标，我们将首先根据工业合作伙伴多年来获得的数据库构建3D对象及其风格化纹理图的数据集。然后，我们将体验应用于纹理映射和对象的3D渲染的图像到图像转换GAN。最后，我们将比较这两种方法，并探索未来的方向，以减少对大型训练集的需求。 该应用程序将为加拿大的视频游戏行业做出贡献，该行业在2017年为GDP贡献了37亿美元。美国和美国之间的伙伴关系。拉瓦尔和Gearbox Studio Québec都将利用美国大学研究团队在深度学习和计算机视觉方面的优势。拉瓦尔和齿轮箱在视频游戏开发方面的专业知识。特别是，工业合作伙伴拥有过去游戏制作中使用的数千个风格化3D模型的独特数据集，这些数据集将在这项工作中得到利用。

Video games, especially large-scale productions such as AAA games, contain thousands of 3D assets that have been crafted by tens or even hundreds of highly skilled artists. In order to give the game a desired artistic look, these artists first model the assets in 3D and assign them default texture. Then, they proceed to carefully "paint" additional stylistic details to the texture in order to give them a desired look. This process, commonly known as "inking", is tedious, time-consuming and must sometimes be repeated several times when the artistic direction changes. In this grant application, we propose to develop computer vision algorithms for automatically applying a desired style onto the base texture map of 3D objects, and as such reproduce the inking process done by artists. The algorithms must be automatic, should be able to learn the stylization automatically from examples, and should be generic enough to be applied to thousands of different 3D models. Our key idea is to rely on recent image-to-image translation approaches based on deep generative adversarial networks (GANs). To attain these objectives, we will first build a dataset of 3D objects and their stylized texture maps based on the database acquired by the industrial partner over the years. Then, we will experiemnt with image-to-image translation GANs applied to texture maps, and to 3D renders of the object. Finally, we will compare both approaches, and explore future directions to reduce the need for large training sets. This application will contribute to the video game industry in Canada, which contributed $3.7B in GDP in 2017. The partnership between U. Laval and Gearbox Studio Québec will both exploit the strengths in deep learning and computer vision of the research team at U. Laval and the expertise in video game development at Gearbox. In particular, the industrial partner possesses a unique dataset of thousands of stylized 3D models used in past game productions which will be leveraged in this work.