CAREER: Natural User Interfaces for Children

自然的用户界面允许用户通过触摸、手势和运动等方式与技术互动。它们是实现普适计算愿景的关键要素，但它们对支持儿童提出了挑战。PI对儿童触摸屏交互的研究发现，现有的针对成人输入设计、训练和测试的表面手势识别算法，以及基于成人交互模式开发的交互设计指南，并不同样适用于儿童。例如，如果典型的手势界面期望将手势作为单个笔划输入，则系统将不能处理由儿童生成的多个笔画，从而导致该儿童的交互不成功。对全身交互手势的识别经历了类似的挑战；儿童更有可能以更大的强度或不同的运动路径执行一个动作或手势(例如，“跳”或“挥手”)，而不是成年人做同样的手势。在这项研究中，PI的目标是从根本上提高我们对如何为儿童设计和开发自然用户交互的理解。这项研究将分三个阶段进行：(1)数据收集和分析：收集小学年龄儿童在各种通道中的输入行为，并分析儿童输入的模式和特征；(2)识别和分类：开发符合儿童预期输入行为模式的新识别算法，并使用机器学习来评估他们的表现；以及(3)多通道交互：调查儿童表现出的多通道输入模式，并验证多通道合成的新方法，这些方法在儿童的自然输入上表现良好。将开发一个试验床应用程序，以展示教育领域的发现。开源的自然用户交互识别和合成算法以及清晰、实用的设计建议将在同行评审的论文和项目网站上发布，供研究人员和实践者使用。这项研究将在理解儿童与计算机与自然用户界面的交互方面取得根本性进展，并开发识别输入和从错误中恢复的可靠新方法。这项工作将有助于解决交互设计研究问题，例如如何最好地适应和使用这些新的儿童模式，以及机器学习研究问题，例如如何开发适合儿童输入的智能多模式识别算法。这项研究所获得的知识和提供的贡献将为在学习环境中为儿童设计泛在计算提供信息。为此，PI将在触摸屏交互和全身交互两种自然模式下为儿童创建预期输入行为模式的模型，并使用这些模型开发和调整智能识别算法，以处理儿童的输入。多模式交互，即对来自多个同时和非同步输入流的不同输入的简化处理，也是自然用户界面的关键组件。像单模式识别一样，成人输入的多模式融合的传统方法可能也不适用于儿童。具体地说，以下研究问题是有针对性的：儿童在多通道自然用户输入通道中产生交互行为的方式是什么？对于使用多模式自然用户输入模式的儿童来说，哪些交互设计技术最有效？针对自然用户交互模式的新的多模式识别和融合算法在儿童输入上有哪些有效的表现？这项工作将与学校和教师合作进行，使研究结果对真实儿童如何在教育环境中使用自然用户交互技术产生直接影响。该项目将包括本科生和研究生研究助理，通过招募女性和代表性不足的少数族裔学生，扩大对计算机科学的参与。教育计划的重点是开发新的以人为中心的计算本科证书。

Natural user interfaces allow users to interact with technology through modalities like touch, gesture, and motion. They are a key element in realizing the vision of ubiquitous computing, yet they present challenges with respect to supporting children. The PI's research on touchscreen interaction for children has found that existing surface gesture recognition algorithms designed, trained, and tested on adult input, and interaction design guidelines developed based on adult interaction patterns, do not apply equally well to children. For example, if a typical gesture interface is expecting a gesture to be entered as a single stroke, the system will not be able to process the multiple strokes generated by a child, leading to an unsuccessful interaction for that child. Recognition for whole-body interaction gestures experiences similar challenges; a child is more likely to perform an action or gesture (e.g., "jump" or "wave") with greater intensity or different motion paths than an adult performing the same gesture. In this research the PI's goal is to fundamentally advance our understanding of how to design and develop natural user interactions for children. The research will be carried out in three phases: (1) Data Collection and Analysis: collection of input behaviors from elementary-school aged children in each modality, and analysis for patterns and characteristics of children's input; (2) Recognition and Classification: development of new recognition algorithms attuned to the expected input behavior patterns of children and use of machine learning to evaluate their performance; and (3) Multimodal Interaction: investigation of multimodal input patterns exhibited by children, and validation of new approaches to multimodal synthesis that perform well on children's natural input. A testbed application will be developed to showcase the findings in an educational domain. Open-source natural user interaction recognition and synthesis algorithms and clear, practicable design recommendations will be developed and released in both peer-reviewed papers and on the project website for use by researchers and practitioners.This research will make fundamental advances in our understanding of child-computer interaction with natural user interfaces and develop robust new approaches for recognizing input and recovering from errors. This work will contribute solutions to interaction design research questions, such as how to best adapt and use these new modalities for children, and machine learning research questions, such as how to develop intelligent multimodal recognition algorithms tailored for children's input. The knowledge gained and contributions delivered by this research will inform the design of ubiquitous computing for children in learning contexts. To these ends, the PI will create models of expected input behavior patterns for children in two natural modalities, touchscreen interaction and whole-body interaction, and use these models to develop and adapt intelligent recognition algorithms tailored to process children's input. Multimodal interaction, or streamlined processing of disparate input from multiple simultaneous and unsynchronized input streams, is also a key component of natural user interfaces. Like unimodal recognition, traditional approaches to multimodal fusion for adult input may also not apply well to children. Specifically, the following research questions are targeted: What are the ways children produce interaction behaviors in multimodal natural user input modalities? What interaction design techniques are most effective for children using multimodal natural user input modalities? What new multimodal recognition and fusion algorithms for natural user interaction modalities perform effectively on children's input? This work will be conducted in partnership with schools and teachers, allowing the research findings to have immediate impact on how real children are using natural user interaction technology in educational contexts. The project will involve both undergraduate and graduate research assistants, broadening participation in computer science by recruiting women and underrepresented minority students. Education plans focus on developing a new undergraduate certificate in Human-Centered Computing.