Verbal/nonverbal bidirectional remote dialog system performed by amicable agent to man

本研究的目的是使用户能够使用机器提供的功能和内容，而不是通过交谈语言和非语言信息，包括人与人之间交换的手势/手势，而不是当前主流的鼠标和键盘，来使用机器提供的功能和内容。当我们与其他人交谈时，我们通常共享对话环境。在本研究中，为了实现网上购物等虚拟环境中虚拟主体(化身)与用户之间的实时对话，利用大阪大学八木教授开发的硅并行视觉芯片和SCRAM网络制作即时传输…，实现了包括语言和视觉信息处理、语音和图形生成在内的并行计算计算机之间可能有更多的信息。当用户的手停下来的时候，当他的食指指向他想要指定的目标对象时，他可以用硅平行视觉芯片捕捉到他的手。另一方面，在语音识别中，在语音识别处理的第一阶段中检测到诸如‘That’或‘This’之类的指示代词，然后将其发出的时间与进行指向动作时的时间进行比较。通过SCRAM网络对它们进行比较，实现了言语和动作的同步识别，并识别出了用指向动作指定的对象。用户的话语可以通过用宾语名称替换句子中的指示代词来分析，对应于用户需求的答案由化身通过话语和行为来表达。通过用户的中断操作来解决通过指向动作指定的目标对象识别的模糊性；当化身在识别目标对象时出错时，可以通过用户的操作来修正化身的行为，所述操作意味着中断、禁止或拒绝。较少

The aim of this research is to make it possible for users to use functions and contents machineries offer without suffering from operation of complicated machineries by talking verbal and non-verbal information including gesture/a hand gesture exchanged between persons as interface between a system and person instead of mouse and keyboard which are the current mainstream. When we are talking with other people, we usually share the dialog environment. And in such a case not only language but also gesture/hand gesture and facial expressions are utilized to convey contents which we want to express each other.In this research, to realize a dialog in real time between a virtual agent (avatar) and a user in the virtual environment such as Internet shopping, parallel computation including linguistic and visual information processing, voice and graphic generation was realized using silicon parallel vision chips developed by Prof. Yagi of Osaka University and SCRAM NET making instant transmissi … More on of information between computers possible.A user's hand can be captured with a silicon parallel vision chip as his hand stops at the moment when he is going to point his forefinger at a target object he wants to specify. Then it is possible to detect instantly whether a user is pointing his finger at the target object or not by only capturing the region including only a hand with a CCD camera.On the other hand, in voice recognition the demonstrative pronoun such as 'that' or 'this' is detected at the first phase of voice recognition process, then the time when it is uttered is compared with that when the pointing action is conducted. By comparing them through SCRAM NET, synchronized recognition between utterance and action is attained and the object specified with a pointing action is identified. User's utterance can be analyzed by substituting the object name for the demonstrative pronoun in the uttered sentence.The answer corresponding to user's demand is expressed using utterance and behavior by the avatar. The ambiguity in identification of a target object specified with a pointing action is solved by interrupt operation by a user ; when an avatar made a mistake in identifying the target object, the avatar's behavior the can be revised by user's operation which means interruption, prohibition or denial. Less