Construction of Multimodal Emotion Representation Model for Computer Animation

为了阐明情感是如何在语音中出现的，我们使用了一种最自然、最有情感的语音数据--落语漫画语音数据进行了分析。因此，语音的情感变异主要表现在话语的末尾。然后重点研究了笑声作为一种情绪表征的生理功能。分析结果表明，f0频率和音素时序是感知语音为笑的基本特征。然后，为了从语言指令中生成带有情感的动作，我们构建了情感表示模型，其中情感词与动作之间的关系被描述为一棵二叉树。然后我们实现了虚拟演员系统，该系统包括情感表示组件，用于使用情感表示模型从语言指令中生成目标运动，以及情感学习组件，用于在给定未知词时更新情感表示模型。最后，为了阐明视频和声音信号与我们从中感知到的情感之间的关系，我们使用带有情感的视觉和语音数据进行了分析。因此，说话人通过声音而不是面部表情的变化来表征情绪水平。同时，听者从说话人的面部表情中识别出说话人的情感类型，并从说话人的声音中感知出说话人的情感程度。

To clarify how emotion appears in speech sound, we analyzed it using rakugo comic stories speech data, which is a kind of the most natural and emotional speech data. As a result the variance in speech sound with emotion mainly appears at the end of utterance. Then we focused on laughing voice as an emotion representation in physiological function. As a result of its analysis f0 frequency and phoneme timing are the fundamental features to perceive the voice as laughing.Then to generate motion with emotion from language instructions we constructed emotion representation model, in which the relation between the emotional words and the motion is described as a binary tree. Then we implemented the virtual actor system, which consists of the emotion representation component to generate target motion from language instructions using the emotion representation model, and the emotion learning component to update the emotion representation model when an unknown word is given. As a result of the evaluation experiment our system generates the appropriate motion with emotion.Finally, in order to clarify the relation between the signals of video and sound and the emotion which we perceive from them, we analyzed it using the visual and speech data with emotion. As a result speakers represent emotional level by the change of not facial expression but voice. At the same time listeners recognize the kind of emotion from the speakers' facial expression, and perceive the level of the emotion from the speakers' voice.