Research on the realization of both integration and synchronization of visual, auditory and hap tic sensation in virtual environment and its applicatiion

成功地实现了本研究的以下目标。(1)产生视觉感知与听觉触觉同步的虚拟空间；(2)在虚拟空间中活动的数字人（虚拟化身）的动作和话语同步，完成人与虚拟化身的语言/非语言交流系统。(1)实现高速运动的虚拟物体与静态虚拟物体的实时碰撞检测；碰撞检测是使用虚拟对象的oct树表示实现的。该系统实现了碰撞噪声与碰撞产生的反作用力同步，并将力输出到一个名为PHANToM的力反馈设备。作为一个具体的例子，成功地实现了一个具有绘图的实验系统，当鼓或铙钹被棍棒敲击时产生的声音和触觉同步。对10名考生进行了三种感觉同步程度的问卷调查。结果表明，鼓槌撞击鼓面的位置与反作用力返回的位置相差不大，证实了视觉、听觉和触觉同步的实现。(2)在人与人之间的对话中，对对方的言语或手势表示赞同或理解的表达，以及对对方的言语或行为进行修改以应对对方的打断，对于双方的对话促进是必不可少的。考虑到这些事实，在系统中引入了以下三个组成部分：对人的话语和行为的同时识别，化身的动作和话语的同时生成，以及在话语或行为过程中防止中断的即时处理机制。当戴着磁传感器和数据手套的人对指向物体的虚拟角色说“把这个和这个放在这里”时，虚拟角色就能成功地看着指向的物体点头并完成命令的任务。从而达到了研究的预期目的。少

The following goals of the research are successfully achieved.(1) The generating of virtual space in which the visual sensation is synchronized with the auditory and haptic sensation(2) Synchronization of the action and utterance by a digital human (avatar) acting in virtual space, and the completion of a verbal / non-verbal communication system between a person and an avatar,Regarding (1), in order to realize the real time collision detection between a virtual object moving at high speed and a static virtual object, the collision detection is implemented that uses the oct tree representation of a virtual object. The system is implemented that synchronizes the collision noise with the reaction force occurring with collision and that outputs the force to a force feedback device called PHANToM. As a concrete example, an experimental system that has the drawing, the sound occurred when a drum or cymbals are hit with a stick and haptic sensation synchronized is successfully realized. The e … More stimation is done for ten examinees using a questionnaire asking the degree of the synchronization among the three kinds of senses. As a result, there is the little difference between the position at which a drumstick hits a drum and the place where the reaction force is returned with a PHANToM, realization of a synchronization among the visual, auditory and haptic sense was confirmed.(2) In a dialog between persons, it is understood that the expression to show the agreement or understanding for the utterance or gesture of a partner, and the modification of the utterance or behavior to cope with the interruption from a partner are indispensable to the mutual dialog promotion. Considering these facts, the following three components are introduced into the system: the simultaneous recognition of the utterance and behavior of a people, the simultaneous generation of action and utterance by an avatar, and the immediate processing mechanism against the interruption during the utterance or behavior. When a people wearing a magnetic sensor and data glove tells "put this and this here" to the avatar pointing the objects, the avatar can successfully nod looking at the object pointed and complete the task ordered.Consequently the expected aim of the research was achieved. Less