CAREER: Natural User Interfaces for Children

自然用户界面允许用户通过触摸、手势和动作等方式与技术进行交互。它们是实现无处不在的计算愿景的关键因素，但是它们在支持儿童方面提出了挑战。PI对儿童触摸屏交互的研究发现，现有的针对成人输入设计、训练和测试的表面手势识别算法，以及基于成人交互模式开发的交互设计指南，并不同样适用于儿童。例如，如果一个典型的手势界面期望一个手势作为一个笔画输入，那么系统将无法处理一个孩子生成的多个笔画，从而导致该孩子的交互不成功。对全身互动手势的识别也面临着类似的挑战；与成年人相比，儿童更有可能以更大的强度或不同的运动路径执行一个动作或手势（例如，“跳跃”或“波浪”）。在这项研究中，PI的目标是从根本上推进我们对如何为儿童设计和开发自然用户交互的理解。本研究将分三个阶段进行：(1)数据收集与分析：收集小学学龄儿童在各模态下的输入行为，分析儿童输入的模式和特征；(2)识别和分类：开发适应儿童预期输入行为模式的新识别算法，并使用机器学习来评估他们的表现；(3)多模态交互：调查儿童表现出的多模态输入模式，并验证在儿童自然输入上表现良好的多模态综合新方法。将开发一个测试平台应用程序来展示教育领域的发现。开源的自然用户交互识别和合成算法以及清晰、可行的设计建议将在同行评议的论文和项目网站上发布，供研究人员和实践者使用。这项研究将使我们对儿童计算机与自然用户界面交互的理解取得根本性进展，并开发出识别输入和从错误中恢复的强大新方法。这项工作将为交互设计研究问题提供解决方案，例如如何最好地适应和使用这些新的儿童模式，以及机器学习研究问题，例如如何开发适合儿童输入的智能多模式识别算法。本研究所获得的知识和贡献将为在学习环境中为儿童设计无处不在的计算提供信息。为此，PI将以两种自然模式（触摸屏交互和全身交互）为儿童创建预期输入行为模式模型，并使用这些模型开发和适应智能识别算法，以处理儿童的输入。多模式交互，或对来自多个同步和非同步输入流的不同输入进行简化处理，也是自然用户界面的关键组成部分。与单模态识别一样，成人输入的传统多模态融合方法也可能不适用于儿童。具体而言，以下研究问题是有针对性的：儿童在多模态自然用户输入模式下产生交互行为的方式是什么？对于使用多模态自然用户输入方式的儿童，什么交互设计技术是最有效的？对于自然用户交互模式，有哪些新的多模态识别和融合算法可以有效地处理儿童的输入？这项工作将与学校和教师合作进行，使研究结果对真正的儿童如何在教育环境中使用自然用户交互技术产生直接影响。该项目将涉及本科生和研究生的研究助理，通过招募女性和代表性不足的少数民族学生，扩大对计算机科学的参与。教育计划的重点是开发一种新的以人为本的计算机本科证书。

Natural user interfaces allow users to interact with technology through modalities like touch, gesture, and motion. They are a key element in realizing the vision of ubiquitous computing, yet they present challenges with respect to supporting children. The PI's research on touchscreen interaction for children has found that existing surface gesture recognition algorithms designed, trained, and tested on adult input, and interaction design guidelines developed based on adult interaction patterns, do not apply equally well to children. For example, if a typical gesture interface is expecting a gesture to be entered as a single stroke, the system will not be able to process the multiple strokes generated by a child, leading to an unsuccessful interaction for that child. Recognition for whole-body interaction gestures experiences similar challenges; a child is more likely to perform an action or gesture (e.g., "jump" or "wave") with greater intensity or different motion paths than an adult performing the same gesture. In this research the PI's goal is to fundamentally advance our understanding of how to design and develop natural user interactions for children. The research will be carried out in three phases: (1) Data Collection and Analysis: collection of input behaviors from elementary-school aged children in each modality, and analysis for patterns and characteristics of children's input; (2) Recognition and Classification: development of new recognition algorithms attuned to the expected input behavior patterns of children and use of machine learning to evaluate their performance; and (3) Multimodal Interaction: investigation of multimodal input patterns exhibited by children, and validation of new approaches to multimodal synthesis that perform well on children's natural input. A testbed application will be developed to showcase the findings in an educational domain. Open-source natural user interaction recognition and synthesis algorithms and clear, practicable design recommendations will be developed and released in both peer-reviewed papers and on the project website for use by researchers and practitioners.This research will make fundamental advances in our understanding of child-computer interaction with natural user interfaces and develop robust new approaches for recognizing input and recovering from errors. This work will contribute solutions to interaction design research questions, such as how to best adapt and use these new modalities for children, and machine learning research questions, such as how to develop intelligent multimodal recognition algorithms tailored for children's input. The knowledge gained and contributions delivered by this research will inform the design of ubiquitous computing for children in learning contexts. To these ends, the PI will create models of expected input behavior patterns for children in two natural modalities, touchscreen interaction and whole-body interaction, and use these models to develop and adapt intelligent recognition algorithms tailored to process children's input. Multimodal interaction, or streamlined processing of disparate input from multiple simultaneous and unsynchronized input streams, is also a key component of natural user interfaces. Like unimodal recognition, traditional approaches to multimodal fusion for adult input may also not apply well to children. Specifically, the following research questions are targeted: What are the ways children produce interaction behaviors in multimodal natural user input modalities? What interaction design techniques are most effective for children using multimodal natural user input modalities? What new multimodal recognition and fusion algorithms for natural user interaction modalities perform effectively on children's input? This work will be conducted in partnership with schools and teachers, allowing the research findings to have immediate impact on how real children are using natural user interaction technology in educational contexts. The project will involve both undergraduate and graduate research assistants, broadening participation in computer science by recruiting women and underrepresented minority students. Education plans focus on developing a new undergraduate certificate in Human-Centered Computing.