Gestural Interaction Paradigms for Smart Spaces

徒手手势在普适计算环境下的交互中具有巨大的潜力，并可能成为未来除触摸之外的主要交互模式。虽然3D手势交互已经探索了很多年，但关于其句法结构和在不同普适计算环境中的适用性的科学知识仍然是零散的，并且局限于特定的有限领域。这使得普适计算环境的交互设计者很难决定在哪些情况和环境中应该使用哪些特定的手势。为了缓解这个问题，这个建议的目的是了解在普适计算环境中的手势的共同特点，以详细说明他们的“成功”的因素，跨情况和域。基于观察，强大的技术来识别手势的行为水平将很快广泛提供（通过各种传感器，例如深度相机），我们的目标是开发和评估模型，涵盖所有相关的语法和语义方面的手势，可以被认为是相关的成功应用的手势在普适计算环境。为此，我们计划开发一个通用的手势词汇表和手势语法，概括了域。这些将基于两个领域的大规模启发式研究：智能家居和智能零售环境，这将导致一个大型的手势语料库进行进一步分析。一个新的手势符号将被提供，以帮助注释语料库和再现徒手手势进一步评估的目的。预期的手势模型将基于四个维度，涉及与徒手手势相关的社会，空间，身体和认知需求。在项目过程中，将制定相应的指标，以便对手势进行信息比较。这些指标也将有助于注释语料库。我们的研究结果将在大规模的现场和实验室研究中得到经验验证。作为该项目的进一步成果，我们计划提供一个软件套件，允许其他研究人员运行和复制我们的研究。在这个项目中，我们将开发并相继使用软件套装的各个模块，并使我们的手势语料库可用。这将包括开发一个手势符号编辑器，该编辑器将用于注释语料库，一个注释编辑器，该编辑器将促进沿着上述四个维度的手势注释，一个语法编辑器，用于为手势序列提供语义，以及一个评估工具包，用于支持使用给定手势语料库的在线用户研究。这些工具将被集成到一个工具链中，不仅在项目中使用，而且向社区公开。

Freehand gestures have great potential to shape the interaction with pervasive computing environments and might constitute, beside touch, the main interaction paradigm in the future. While 3D gestural interaction has been explored for many years now, the scientific knowledge on their syntactical structure and the applicability across different pervasive computing environments is still fragmented and bound to specific confined domains. This makes it very difficult for interaction designers of pervasive computing environments to decide which specific gestures should be used in which situations and environments. To alleviate this problem, this proposal aims to understand the common characteristics of gestures in pervasive computing environments to elaborate on factors of their “success” across situations and domains. Based on the observation that robust techniques to recognize gestures on a behavioural level will be soon broadly available (through various sensors, e.g. depth-cameras), we aim to develop and evaluate models that cover all relevant syntactic and semantic aspects of gestures that can be considered relevant for the successful application of gestures in pervasive computing environments. For this purpose we plan to develop a general vocabulary of gestures and a gesture grammar that generalizes over domains. These will be based upon a large-scale elicitation study in two domains: a smart home and an intelligent retail environment that will result in a large gesture corpus for further analysis. A novel gesture notation will be provided to help annotate the corpus and to reproduce freehand gestures for further evaluation purposes. The intended gestural model will be based on four dimensions, related to the social, spatial, physical and cognitive demands associated with freehand gestures. In the course of the project corresponding metrics will be developed to allow for an informative comparison of gestures. The metrics will also help to annotate the corpus. Our findings will be validated empirically in large-scale field as well as lab studies. As a further result of the project, we plan to provide a software suite that allows other researchers to run and replicate our studies. During the project we will develop and successively use individual modules of the software suit and make our elicited and annotated gesture corpus available. This will include the development of a gesture notation editor, which will be used to annotate the corpus, a annotation editor that will facilitate annotation of gesture along the four dimensions described above, a grammar editor to provide semantics to gesture sequences, and an evaluation toolkit to support online user studies with a given gesture corpus. The tools will be integrated into a tool chain, that will be not only used within the project, but made publicly available to the community.