Experimental Study on Human-Machine Interfaces Based on Real-time Close Interaction

基于视觉的人机界面从视频图像中识别手势和凝视等非语言行为引起了人们的极大兴趣。然而，传统的这种界面对用户施加了几个不友好的限制。此外，他们有时也不被认可。一旦失败，他们中的大多数人将永远无法从失败中恢复过来。在本项目中，我们提出了通过利用用户与机器之间的实时近距离交互来解决这些问题，并开发了实验系统来验证其有效性。首先，我们放宽了对用户位置的限制，使用主动摄像头跟踪用户。然后，我们放宽了对用户无意手部动作的限制。常规系统可能错误地将这样的动作识别为意图发出命令的手势。我们使用另一种非语言行为，脸部方向，来只选择有意的动作。这里的假设是，当用户想要对目标对象进行操作时，他/她正在观察目标对象。最后，我们提出使用语音和动作的交互来识别未知手势，并恢复识别失败。当系统对其识别结果不确定时，它通过语音和/或动作来显示当前情况，这是机器的非语言行为。然后它观察用户的反应。实验结果表明，通过迭代这种交互，我们最终可以将我们的意图传递给系统。通过开发一个可以在虚拟世界中通过手势移动对象的系统，我们证明了该方法的有效性。此外，我们还开发了一种智能轮椅，以检查这种非语言界面是否适用于在现实世界中工作的系统。我们使用脸部方向来控制轮椅。我们可以通过朝我们想去的方向看来转向它。实验结果证实这是一种很好的接口方式。

Vision-based human interfaces recognizing nonverbal behaviors such as gestures and gaze from video images have attracted great interest. However, conventional such interfaces impose several unfriendly restrictions on the users. In addtion, they sometimes fail in their recognition. And once they fail, most of them will never recover from the failure. In this project, we have proposed to solve these issues by exploiting real-time close interaction between the user and the machine, and developed experimental systems to verify the effectiveness.First, we have relaxed restrictions on the user's position using active cameras to track the user. Then, we have eased restrictions on the user's unintentional hand motions. Conventional systems may erroneously recognize such motions as gestures intended to give commands. We use another nonverbal behavior, face direction, to choose only intentional motions. The assumption here is that the user is watching the target object when he/she wants to operate on it. Finally, we have proposed to use interaction by speech and action to recognize unknown gestures and to recover recognition failures. When the system is not certain of its recognition results, it shows the currentsituation by speech and/or action, which is the nonverbal behavior for the machine. Then it observes the user's reaction. Experimental results show that we can eventually pass our intention to the system by iterating such interaction.We have demonstrated the effectiveness of our approach by developing a system in which we can move objects in a virtual world by hand gestures. In addition, we have developed an intelligent wheelchair to examine whether or not such nonverbal interfaces are applicable to systems working in the real world. We use face direction to control the wheelchair. We can turn it by looking in the direction where we want to go. Experimental results confirm this as a good means of interface.