Modeling Diverse, Personalized and Expressive Animations for Virtual Characters through Motion Capture, Synthesis and Perception

在游戏开发、机器人技术、虚拟现实(VR)和增强现实(AR)应用中，角色动画在为虚拟角色传递运动方面发挥着关键作用。尽管在将动作建模为自然和逼真方面做出了大量努力，但之前的研究一直集中在一般动作内容学习上，其中每个人如何执行内容的风格特征在很大程度上被忽略了。如果没有单独的风格，不同的虚拟角色在需要相同的动作内容时会以相同的方式移动，这远远不能满足创造一个多样化的虚拟世界的要求。样式化运动的一个主要挑战是缺乏大规模的运动样式数据库，因此没有足够的关于如何有效地建模和传递样式的知识。这项建议侧重于动作风格的学习、合成和迁移。从长远来看，我们的目标是找到最终的解决方案，以生成与现实世界中的多样性相匹配的变化的风格化动作。我们的短期目标是：首先通过运动捕捉技术建立大规模的运动风格数据库；从这些数据库中，我们开发方法来学习有效的运动表示，并探索生成模型来有条件地生成所需样式的动作；我们将进一步发现样式转换模型，以便在保持原始动作内容的同时编辑样式。在五年期间，我们将具体捕捉、学习和示范三种文体特征：属于不同人群的人口统计风格，如年龄、性别和种族；因个性而产生的个性化风格，不同人的体型；表现风格，展示同一个人在不同情景下不同的情感和身体状态。我们将建立涵盖这些风格变化的数据库，公开数据库以供开放使用，并向公众提供标签、文件和技术支持。研究成果和源代码将被公布，解决风格特征的提取，以可控的方式生成动作风格，以及将风格转换为新动作的问题。在五年任期之后，我们将继续向我们的数据库添加更多样式，以模拟更广泛的运动样式。我们的数据库可以直接用于AR/VR和游戏场景中的各种角色的动画制作，它也有助于其他研究人员对动作风格进行建模，并促进心理学、运动学和艺术学等学科的交叉研究。动作风格建模方面的研究成果支持动作输入的智能应用，如动作识别、风格识别和人物识别。风格合成和传递技术也可以广泛应用于游戏、娱乐行业、教育、媒体和社交网络的AR/VR应用。通过创造真实世界中真实体现人的虚拟角色，这项工作可以促进虚拟世界的多样性、公平性和包容性。

Character animation plays a key role in delivering motions for virtual characters in game development, robotics, Virtual Reality (VR) and Augmented Reality (AR) applications. Despite the large effort in modeling motions to be natural and realistic, previous research has been focused on generic motion content learning where stylistic features of how each individual performs the content are largely ignored. Without individual styles, different virtual characters move the same way when the same motion content is needed, which is far from satisfactory to create a diverse virtual world. One major challenge to stylize motions is the lack of large scale motion style databases, and thus no sufficient knowledge of how to effectively model and transfer styles. This proposal focuses on motion style learning, synthesis and transfer. In the long term, our goal is to find ultimate solutions to generate stylized motions with variations that match the diversity in the real-world. The short-term objectives are: we will first establish large scale motion style databases through motion capture technique; from the data, we develop methods to learn effective motion representations and explore generative models to conditionally generate motions with desired styles; we will further discover style transfer models to edit styles while keeping the original motion content. In the five-year period, we will specifically capture, learn and model three stylistic features: demographic styles belonging to different groups of people, e.g. age, gender, and race; personalized styles resulting from personalities, body build for different individuals; expressive styles demonstrating varied emotional and physical states for the same individual under different scenarios. We will set up our databases to cover these style variations, publicize the database for open access, and provide labelling, documentation and technical support to the public. Research findings and source code will be published, solving problems of extracting style features, generating motion styles in a controllable manner, and transferring styles to novel motions. Beyond the five-year term, we will continue adding more styles to our databases, to model a broader picture of motion styles. Our database can directly be used in animating diverse characters in AR/VR and game scenes, it also facilitates other researchers to model motion styles, and stimulates interdisciplinary research in psychology, kinesiology, and art etc. Research findings in motion style modeling supports intelligent applications such as action recognition, style recognition and person identification from motion input. Style synthesis and transfer technology can also be widely applied in the game, entertainment industry, AR/VR applications in education, media, and social networks. By creating virtual characters that authentically embody people in the real world, this work can promote Diversity, Equity and Inclusion of the virtual world.