Verbal/nonverbal bidirectional remote dialog system performed by amicable agent to man

本研究的目的是使用户能够通过口头和非口头信息（包括人与人之间交换的手势/手势）作为系统与人之间的界面，而不是当前主流的鼠标和键盘，来使用机器提供的功能和内容，而无需忍受复杂机器的操作。当我们与其他人交谈时，我们通常共享对话环境。在这种情况下，不仅使用语言，而且还利用手势/手势和面部表情来传达我们想要表达的内容。在这项研究中，为了在网上购物等虚拟环境中实现虚拟代理（化身）和用户之间的实时对话，使用大阪大学八木教授开发的硅并行视觉芯片和 SCRAM NET 实现了包括语言和视觉信息处理、语音和图形生成在内的并行计算，从而使计算机之间的信息即时传输成为可能。用户的手可以被捕获当他将食指指向他想要指定的目标物体时，他的手会停止，并带有硅平行视觉芯片。然后，只需用 CCD 摄像头捕获仅包括手的区域，就可以立即检测到用户是否将手指指向目标对象。 另一方面，在语音识别中，在语音识别过程的第一阶段检测指示代词，例如“那个”或“这个”，然后将说出该声音的时间与进行指向动作的时间进行比较。通过 SCRAM NET 进行比较，实现言语和动作之间的同步识别，并识别出指向动作指定的对象。通过将说出的句子中的指示代词替换为对象名称，可以分析用户的话语。通过虚拟人物的话语和行为来表达与用户需求相对应的答案。通过用户的中断操作解决了指向动作指定的目标对象标识的模糊性；当虚拟形象在识别目标对象时出现错误时，虚拟形象的行为可以通过用户的操作进行修正，即中断、禁止或拒绝。较少的

The aim of this research is to make it possible for users to use functions and contents machineries offer without suffering from operation of complicated machineries by talking verbal and non-verbal information including gesture/a hand gesture exchanged between persons as interface between a system and person instead of mouse and keyboard which are the current mainstream. When we are talking with other people, we usually share the dialog environment. And in such a case not only language but also gesture/hand gesture and facial expressions are utilized to convey contents which we want to express each other.In this research, to realize a dialog in real time between a virtual agent (avatar) and a user in the virtual environment such as Internet shopping, parallel computation including linguistic and visual information processing, voice and graphic generation was realized using silicon parallel vision chips developed by Prof. Yagi of Osaka University and SCRAM NET making instant transmissi … More on of information between computers possible.A user's hand can be captured with a silicon parallel vision chip as his hand stops at the moment when he is going to point his forefinger at a target object he wants to specify. Then it is possible to detect instantly whether a user is pointing his finger at the target object or not by only capturing the region including only a hand with a CCD camera.On the other hand, in voice recognition the demonstrative pronoun such as 'that' or 'this' is detected at the first phase of voice recognition process, then the time when it is uttered is compared with that when the pointing action is conducted. By comparing them through SCRAM NET, synchronized recognition between utterance and action is attained and the object specified with a pointing action is identified. User's utterance can be analyzed by substituting the object name for the demonstrative pronoun in the uttered sentence.The answer corresponding to user's demand is expressed using utterance and behavior by the avatar. The ambiguity in identification of a target object specified with a pointing action is solved by interrupt operation by a user ; when an avatar made a mistake in identifying the target object, the avatar's behavior the can be revised by user's operation which means interruption, prohibition or denial. Less