Experimental Study on Human-Machine Interfaces Based on Real-time Close Interaction

基于视觉的人机界面识别非语言行为（如手势和视频图像中的凝视）引起了人们的极大兴趣。然而，传统的接口对用户施加了一些不友好的限制。此外，他们有时无法识别。一旦他们失败了，他们中的大多数人将永远无法从失败中恢复过来。在这个项目中，我们提出了利用用户与机器之间的实时密切交互来解决这些问题，并开发了实验系统来验证其有效性。首先，我们放松了对用户位置的限制，使用活动摄像头跟踪用户。然后，我们放宽了对用户无意识手势的限制。传统的系统可能会错误地将这些动作识别为意在发出命令的手势。我们使用另一种非语言行为，面部方向，来选择有意的动作。这里的假设是，当用户想要对目标对象进行操作时，他正看着它。最后，我们提出了使用语音和动作的交互来识别未知手势并恢复识别失败。当系统不确定其识别结果时，它通过言语和/或动作来显示当前情况，这是机器的非语言行为。然后观察用户的反应。实验结果表明，通过迭代这种交互，我们最终可以将我们的意图传递给系统。我们已经通过开发一个系统证明了我们方法的有效性，在这个系统中，我们可以通过手势在虚拟世界中移动物体。此外，我们还开发了一种智能轮椅，以检查这种非语言接口是否适用于现实世界中的系统。我们用脸部方向来控制轮椅。我们可以把它转向我们想去的方向。实验结果证实了这是一种很好的界面方法。

Vision-based human interfaces recognizing nonverbal behaviors such as gestures and gaze from video images have attracted great interest. However, conventional such interfaces impose several unfriendly restrictions on the users. In addtion, they sometimes fail in their recognition. And once they fail, most of them will never recover from the failure. In this project, we have proposed to solve these issues by exploiting real-time close interaction between the user and the machine, and developed experimental systems to verify the effectiveness.First, we have relaxed restrictions on the user's position using active cameras to track the user. Then, we have eased restrictions on the user's unintentional hand motions. Conventional systems may erroneously recognize such motions as gestures intended to give commands. We use another nonverbal behavior, face direction, to choose only intentional motions. The assumption here is that the user is watching the target object when he/she wants to operate on it. Finally, we have proposed to use interaction by speech and action to recognize unknown gestures and to recover recognition failures. When the system is not certain of its recognition results, it shows the currentsituation by speech and/or action, which is the nonverbal behavior for the machine. Then it observes the user's reaction. Experimental results show that we can eventually pass our intention to the system by iterating such interaction.We have demonstrated the effectiveness of our approach by developing a system in which we can move objects in a virtual world by hand gestures. In addition, we have developed an intelligent wheelchair to examine whether or not such nonverbal interfaces are applicable to systems working in the real world. We use face direction to control the wheelchair. We can turn it by looking in the direction where we want to go. Experimental results confirm this as a good means of interface.