Human Interfaces through Mutual Assistance between Multiple Sensing Modalities and Actions based on Visual Understanding of the World

我们研究了福利服务机器人的人机界面。人类用户可能会要求机器人获得他们需要的东西。语音接口可能最适合这样的机器人。然而，机器人需要视觉来理解人类的话语并执行命令。如果我们确信听众分享了一些视觉信息，我们经常在话语中忽略或模棱两可地提到有关它的事情。我们提出了一种利用计算机视觉来理解具有这种歧义的语音的方法。它跟踪人的凝视方向，检测方向上的物体。它还能识别人类的行为。基于这些视觉信息，它可以理解人的含蓄话语。实验结果表明，该方法有助于实现人性化的语音界面。对于用户友好的界面，还需要对非语言行为进行视觉识别。在这个项目中，我们致力于眼神交流。眼神交流是控制人类交流的一种有效手段，例如在开始交流时。如果我们看着对方，我们似乎可以进行眼神交流。然而，单凭这一点是不能完成眼神交流的。此外，我们需要意识到被对方注视着。考虑到这两种情况，我们提出了一种主动眼神接触的人-机器人交流方法。机器人根据人类的观察结果改变面部表情，进行眼神交流。尽管如上所述，视力很重要，但视力并不总是能正常工作。我们提出了一种交互式的愿景来解决这个问题。当视觉系统无法完成一项任务时，机器人会向用户发出语音，以便用户的自然反应可以为机器人的视觉系统提供有用的信息。

We have investigated human interfaces for welfare service robots. Human users may ask the robots to get something that they need. Speech interfaces may be best suited for such robots. However, the robots need vision to understand utterances of the humans and to carry out the orders. If we are certain that the listeners share some visual information, we often omit or mention ambiguously things about it in our utterances. We have proposed a method of understanding speech with such ambiguities using computer vision. It tracks the human's gaze direction, detecting objects in the direction. It also recognizes the human's actions. Based on these bits of visual information, it understands the human's inexplicit utterances. Experimental results show that the method helps to realize human-friendly speech interfaces. Visual recognition of nonverbal behaviors is also required for user-friendly interfaces. In this project, we have worked on eye contact. Eye contact is an effective means of controlling communication for humans, such as in starting communication. It seems that we can make eye contact if we look at each other. However, this alone cannot complete eye contact. In addition, we need to be aware of being looked by each other. We have proposed a method of active eye contact for human-robot communication considering both conditions. The robot changes its facial expressions according to the observation results of the human to make eye contact. Although vision is important as mentioned, vision may not always work properly. We have proposed an interactive vision to solve this problem. When the vision system cannot achieve a task, the robot makes a speech to the user so that the natural response by the user can give helpful information for the robot's vision system.