Construction of Prototype Multi-modal Interface by Lifelike Agent

构建了多客户端面对面的原型通信系统。在其他客户端的交互驱动下，逼真的代理出现在网络空间中。该系统由一个服务器和多个客户端组成，每个客户端都有摄像头和麦克风。从麦克风捕获的声音逐帧传输到服务器。然后对语音信号进行分析，并通过神经网络将其转换为口型参数。该口型参数通过网络传递给各个客户端，复制另一方客户端的逼真代理的表情和嘴唇运动受此参数控制。语音信号也传输到每个客户端，由每个客户端系统的扬声器与合成的人脸图像同步播放。通过按功能键选择基本表情，可以改变在其他客户端显示的逼真代理的人脸图像。本系统准备了两种模式；行走模式和飞行模式。在行走模式下，每个用户都可以通过眼神交流与他人交流，改变代理眼的位置和方向。在飞行模式下，用户可以作为通信中的观察者。利用该原型通信系统，进行了三个用户相互通信的实验。合成速率为每秒10帧。并且可以实现自然的交流环境。

Prototype communication system between multiple clients in face-to-face style is constructed. Lifelike agent appears in cyberspace driven by other client interactively. This system is composed of one server and multiple clients, and each client has a camera and microphone. Voice captured from microphone is transmitted to server frame by frame. And then voice signal is analyzed and converted to mouth shape parameters by neural network. This mouth shape parameter is transmitted to each client through network and, expression and lip movement of lifelike agent copying other side client are controlled by this parameter. Voice signal is also transmitted to each client and played back by speaker at each client system synchronizing with synthesized face image. Basic expression can be selected by pushing function key to change the face image of lifelike agent displayed at other clients. Two mode is prepared in this system ; walk through mode and fly through mode. In the walk through mode, each user can communicate by eye contact with others and change the location and direction of agent eye. In the fly mode, user can be an observer in communication.By using this prototype communication system, the experiment in which three users communicate each other is performed. Synthesis rate is 10 frames per second. And natural communication environment can be achieved.