Research on the realization of both integration and synchronization of visual, auditory and hap tic sensation in virtual environment and its applicatiion

本研究成功地实现了以下目标。(1)产生视觉与听觉和触觉同步的虚拟空间（2）在虚拟空间中动作的数字人（化身）的动作和话语的同步，以及完成人与化身之间的语言/非语言通信系统，关于（1），为了实现高速移动的虚拟对象与静态虚拟对象之间的真实的实时碰撞检测，使用虚拟对象的OCT树表示来实现碰撞检测。该系统被实现为利用碰撞产生的反作用力消除碰撞噪声，并将力输出到称为PHANToM的力反馈装置。作为一个具体的例子，成功地实现了一个实验系统，该系统具有绘图，当用棍子击打鼓或钹时发生的声音和触觉同步。的e 关于我们 对10名被试进行了三种感觉同步程度的问卷调查。结果，鼓槌击打鼓的位置与通过PHANToM返回反作用力的位置之间几乎没有差异，确认了视觉、听觉和触觉之间的同步的实现。(2)在人与人之间的对话中，应当理解，对伙伴的话语或手势表示同意或理解的表达，以及对话语或行为的修改以科普来自伙伴的中断，对于相互对话的促进是必不可少的。考虑到这些事实，以下三个组成部分被引入到系统中：同时识别的人的话语和行为，同时产生的行动和话语的化身，和即时处理机制，防止中断期间的话语或行为。当穿戴有磁性传感器和数据手套的人对虚拟化身指向的物体说“把这个和这个放在这里”时，虚拟化身就能成功地对着所指向的物体点头并完成任务，从而达到了预期的研究目的。少

The following goals of the research are successfully achieved.(1) The generating of virtual space in which the visual sensation is synchronized with the auditory and haptic sensation(2) Synchronization of the action and utterance by a digital human (avatar) acting in virtual space, and the completion of a verbal / non-verbal communication system between a person and an avatar,Regarding (1), in order to realize the real time collision detection between a virtual object moving at high speed and a static virtual object, the collision detection is implemented that uses the oct tree representation of a virtual object. The system is implemented that synchronizes the collision noise with the reaction force occurring with collision and that outputs the force to a force feedback device called PHANToM. As a concrete example, an experimental system that has the drawing, the sound occurred when a drum or cymbals are hit with a stick and haptic sensation synchronized is successfully realized. The e … More stimation is done for ten examinees using a questionnaire asking the degree of the synchronization among the three kinds of senses. As a result, there is the little difference between the position at which a drumstick hits a drum and the place where the reaction force is returned with a PHANToM, realization of a synchronization among the visual, auditory and haptic sense was confirmed.(2) In a dialog between persons, it is understood that the expression to show the agreement or understanding for the utterance or gesture of a partner, and the modification of the utterance or behavior to cope with the interruption from a partner are indispensable to the mutual dialog promotion. Considering these facts, the following three components are introduced into the system: the simultaneous recognition of the utterance and behavior of a people, the simultaneous generation of action and utterance by an avatar, and the immediate processing mechanism against the interruption during the utterance or behavior. When a people wearing a magnetic sensor and data glove tells "put this and this here" to the avatar pointing the objects, the avatar can successfully nod looking at the object pointed and complete the task ordered.Consequently the expected aim of the research was achieved. Less