Research on Dynamic Visemes of Japanese Speech Utterances

为了动画逼真的嘴唇运动在日本的讲话话语，高速视频录制完成，而一个人的主题说出简短的日语句子。两个高速摄像机捕捉正面和侧面的意见，人类主体的嘴唇在语音平衡的日语短句的讲话。记录速度为每秒300或240个样本，200 × 200或256 × 256像素，24位RGB颜色。在44.1 kHz Mono下记录声信号。除了自然嘴唇之外，对于一些句子记录嘴唇上的标记照明点和嘴唇上的蓝色着色，以使得能够更容易地建模提取嘴唇位置参数。这些录像被编译成日语视位视频数据库，这些视频图像被用来根据MPEG-4人体和人脸参数制作唇点的计算机图形模型参数。特别是对/ba/等爆炸性音素的快速动作进行逐帧观察，实现了嘴唇运动的精确再现，并将这些视位放置在时间轴上的关键帧上，在关键帧之间插入中间帧，完成了计算机图形动画。不仅是静态视位，而且还有动态视位，即，使用一组用于一个视位的多个帧的唇部参数，并使用非线性插值。使用这种方法，更真实的动画嘴唇运动在讲话中实现了比传统的简单插值相对较少的视位。

To animate realistic lip motions during Japanese speech utterances, high-speed video recording was done while a human subject uttered short Japanese sentences. Two high-speed video cameras captured front and side views of the human subject lips during speech utterances of phonemically balanced short Japanese sentences. Recordings were at 300 or 240 samples per second, 200 by 200 or 256 by 256 pixels, 24 bit RGB colors. Acoustic signals were recorded at 44.1 kHz Mono. Other than natural lips, marking illuminated points on lips and blue coloring on lips were recorded for some sentences to enable easier modeling of extracting lip position parameters. These recordings were compiled as a Japanese viseme video database.These video images were used to make computer graphics model parameters of lip points according to MPEG-4 body and face parameters. Especially, fast movements like explosive phonemes like /ba/ were carefully observed frame by frame to realize precise reproductions of lip movements.By placing these visemes onto key frames on time axis and interpolating middle frames between key frames, a computer graphics animation was done. Not only static visemes but also dynamic visemes, i.e., a set of lip parameters for multiple frames for one viseme, were used and nonlinear interpolations were used. Using this method, more realistic animations of lip motions during speech were realized than traditional simple interpolations of relatively small numbers of visemes.