Media Conversion for User-friendly Human-Machine Interface

1992年，我提出了从文本和语音到面部图像的媒体转换系统。唇动由文本和语音的分析结果控制。首先，使用频谱和功率转换提取元音位置的段边界。通过切分结果与文本信息的DP匹配确定严格的元音和辅音位置。该算法实现了基于文本信息的音素精确切分。该系统也可用于语音合成。嘴部动画的关键帧位于由分割结果确定的分割边界上。通过文本分析确定标准口型和时长。关键帧之间的参数由三维样条插值决定，使运动变得平滑自然。通过对三维线框模型的修改实现表达。操作员只需通过在时间轴上定位图标化的面部表情来确定关键帧的位置。利用该系统实现了一个带表情的电子邮件接口原型系统。发送端用户仅给出文本、语音和表情位置。该方法只需传输少量的参数，经过一定的延时，就可以在接收端得到与语音同步的人脸运动图像。该系统可以实现超低码率的图像传输。人脸表情识别是我们未来的研究方向。

In 1992, I proposed media conversion system from both text and voice to facial image. Lip motion is controlled by the analysis results of text and voice. At first, segment boundaries of vowel positions are extracted using spectrum and power transition. Strict vowel and consonant positions are decided by DP matching of segmentation result and text information. By this algorithm, accurate phoneme segmentation using text information is realized. This system can be applied to speech synthesis, too. The keyframes of mouth animation are located on the segment boundary decided by segmentation result. Standard mouth shapes and durations are decided by the text analysis. The parameters between keyframes are decided by 3-D Spline interpolation, so the motion becomes smooth and natural.In 1993, I constructed Scenario Making System to describe facial animation easily. Expression is realized by the modification of 3D wire frame model. Operator only has to decide the keyframes position by locating the Iconized facial expression on the time axis. Using this system, I realized proto-type e-mail interface system with facial expression. User of transmitter side only gives text, voice and expression positions. Only few parameters are transmitted and after a few delay, the facial motion image synchronized with the voice appears at the terminal of receiver. Ultra low bit rate image transmittion can be realized by our system. The facial expression recognition is our future subject.