Prehensile Interaction: User Interaction Concepts based on Prehensile Hand Behavior

虽然交互式3D环境在过去几年中越来越受欢迎，但3D用户界面仍然相当复杂，在许多情况下需要特殊的技能和培训。手势交互一直是三维用户界面领域的研究热点，但在设计手界面时，很少考虑到心理学相关领域的研究。在心理学的各个领域对伸手抓握动作的研究一致表明，抓握的自然运动学可以预测一个人将要抓住的物体，有时甚至可以预测随后对该物体采取的行动。许多研究调查了影响抓取的不同因素，例如，物体的大小或形状，其特定的可视性或预期用途，以及障碍物和干扰物的影响。一些研究还评估了在使用实体、记忆或虚拟物体时，握力行为的差异。此外，由于在自然抓取中手指的动力学高度相关，因此可以大大降低抓取运动学的计算复杂度。这些见解为大幅改善3D环境中的交互提供了巨大的机会，前提是手的运动学信息可以在飞行中提取和评估，并立即整合到界面中。在本项目中，我们将建立一个在人机交互背景下分析可抓握行为的一般框架，并将探索可抓握信息在设计与计算机生成的虚拟环境交互的自然用户界面中的适用性。我们计划进行一系列实验，在不同的应用场景、不同的任务和不同的方法下评估用户的手部运动学。目的是收集足够的数据，开发有效的交互式可视化方法来分析它们，并检查基于手部运动学的预期对象和任务的早期预测的优点和局限性。预期的洞察力将使我们能够探索基于自然手部交互的未来交互系统的设计空间，即不需要特殊的手势或技能。

While interactive 3D environments have gained increasing popularity in the last years, 3D user interfaces still remain rather complex and in many cases require special skills and training. Interaction with hand gestures has been an active research area in the 3D user-interfaces field, but research from the related fields of psychology has rarely been taken into consideration when designing the hand-based interfaces. Investigations on reach-to-grasp actions in various domains of psychology have consistently shown that the natural kinematics of prehension allows for predicting the object a human is going to grab and sometimes even the subsequent actions taken on that object. Many studies investigated different factors affecting the prehension, e.g., the object's size or shape, its specific affordances or intended use, as well as the effect of obstacles and distractors. Some studies also evaluated differences in prehensile behavior when using physical, memorized, or virtual objects. Furthermore, it was shown that the computational complexity of grasp kinematics can be greatly reduced, since the fingers' dynamics are highly correlated in natural prehension. These insights promise great opportunities for substantially improved interaction in 3D environments, provided that the hand kinematics information is extracted and evaluated on the fly and instantaneously incorporated in the interface.In this project, we will establish a general framework for the analysis of prehensile behavior in the context of human-computer interaction and we will explore the applicability of prehensile information to design natural user interfaces for the interaction with computer-generated virtual environments. We plan to perform a series of experiments, in which the users' hand kinematics will be evaluated in different application scenarios, for different tasks, and with different methods.The aim is to gather sufficient data, develop effective interactive visual methods to analyze them, and examine the benefits and limitations of early prediction of intended objects and tasks based on hand kinematics. The expected insight will allow us to explore the design space for future interactive systems based on natural hand interactions, i.e., without requiring special gestures or skills.