Deep style transfer for 3D meshes

视频游戏，特别是大型制作，如AAA游戏，包含数以千计的3D资产，这些资产是由数十甚至数百名高技能艺术家精心制作的。为了给游戏带来所需的艺术效果，这些美工人员首先在3D中对资源进行建模，并为其指定默认纹理。然后，他们继续仔细地为纹理绘制额外的风格细节，以便给他们一个想要的外观。这一过程俗称“墨迹”，繁琐、耗时，当艺术方向发生变化时，有时还必须重复几次。 在这项拨款申请中，我们建议开发计算机视觉算法，以自动将所需样式应用到3D对象的基础纹理贴图上，并因此再现艺术家完成的墨迹过程。算法必须是自动的，应该能够从示例中自动学习风格化，并且应该足够通用，以适用于数千个不同的3D模型。我们的主要思想是依赖于最新的基于深度生成对抗网络(GANS)的图像到图像的翻译方法。为了实现这些目标，我们将首先基于工业合作伙伴多年来获得的数据库，建立3D对象及其风格化纹理贴图的数据集。然后，我们将体验应用于纹理贴图和对象的3D渲染的图像到图像转换Gans。最后，我们将比较这两种方法，并探索未来的方向，以减少对大型训练集的需求。 这项应用将为加拿大的视频游戏行业做出贡献，2017年加拿大视频游戏行业为国内生产总值贡献了37亿美元。U·Laval与Gearbox Studio Québec的合作将既利用U·Laval研究团队在深度学习和计算机视觉方面的优势，也利用Gearbox在视频游戏开发方面的专业知识。特别是，工业合作伙伴拥有一个独特的数据集，其中包括在过去的游戏制作中使用的数千个风格化3D模型，这将在本工作中得到利用。

Video games, especially large-scale productions such as AAA games, contain thousands of 3D assets that have been crafted by tens or even hundreds of highly skilled artists. In order to give the game a desired artistic look, these artists first model the assets in 3D and assign them default texture. Then, they proceed to carefully "paint" additional stylistic details to the texture in order to give them a desired look. This process, commonly known as "inking", is tedious, time-consuming and must sometimes be repeated several times when the artistic direction changes. In this grant application, we propose to develop computer vision algorithms for automatically applying a desired style onto the base texture map of 3D objects, and as such reproduce the inking process done by artists. The algorithms must be automatic, should be able to learn the stylization automatically from examples, and should be generic enough to be applied to thousands of different 3D models. Our key idea is to rely on recent image-to-image translation approaches based on deep generative adversarial networks (GANs). To attain these objectives, we will first build a dataset of 3D objects and their stylized texture maps based on the database acquired by the industrial partner over the years. Then, we will experiemnt with image-to-image translation GANs applied to texture maps, and to 3D renders of the object. Finally, we will compare both approaches, and explore future directions to reduce the need for large training sets. This application will contribute to the video game industry in Canada, which contributed $3.7B in GDP in 2017. The partnership between U. Laval and Gearbox Studio Québec will both exploit the strengths in deep learning and computer vision of the research team at U. Laval and the expertise in video game development at Gearbox. In particular, the industrial partner possesses a unique dataset of thousands of stylized 3D models used in past game productions which will be leveraged in this work.