CRII: CHS: Understanding Gesture User Behavior in Augmented Reality Headsets

CRII：CHS：了解增强现实耳机中的手势用户行为

• 批准号: 1948254
• 负责人: Francisco Ortega
• 金额: $ 17.5万
• 依托单位: Colorado State University
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-08-01 至 2024-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1948254&HistoricalAwards=false
• 关键词: CRII; CHS; Understanding; Gesture; User

Augmented Reality (AR) allows placing virtual objects in a real environment. AR glasses are the next wave of human-computer interaction technology. This technology will have a direct impact on education, manufacturing, national defense, and assembly jobs, among others. Understanding the role of human interaction with an AR system using gestures, speech, or the combination of both (gesture and speech) will provide foundational knowledge to decrease the complexity of user interaction. This research will have an impact in society by supporting underrepresented and non-traditional graduate students and developing new interdisciplinary course. The outcomes of this research will facilitate creation of improved AR applications that benefit several fields such as future workforce training. The datasets generated through this project can result in collaborative explorations between computer scientists and cognitive/learning scientists to make learning AR applications more accessible and incorporate more intuitive interactions.This project will generate preliminary, foundational research on gestures for AR three-dimensional (3D) user interfaces, design egocentric gestures with/without speech interactions with AR headsets, construct two labeled datasets for gesture recognition, and develop a block-based, gesture-enabled application for AR headsets. This project will advance the state of the art in gesture interaction for AR headsets by: (1) Elicitation Studies: This project will conduct the largest elicitation study to date, substantially increasing the foundational knowledge base and improving elicitation methodology and gesture recognizers. This study will require creating a novel application that will generate new collaborative interaction datasets, resulting in better foundational knowledge about user gestures for AR 3D user interfaces via gesture elicitation; (2) Dataset Generation: The elicitation studies will produce labeled and unlabeled egocentric datasets that will enable human-computer interaction and computer vision researchers to explore new recognition algorithms; (3) Multi-modal interaction (gesture and speech) interaction in AR Headsets: The preliminary research conducted as part of this project will allow research in complex interactions and collaborative tasks using AR Headsets. This research will include conducting egocentric gesture studies (with/without speech) to determine appropriate interaction for AR headset applications. The project’s research activities will broaden our understanding and the use of elicitation studies beyond action/gesture mapping, as well as the multi-modal interactions (gesture and speech) in AR Headsets, either validating findings in human-to-human communication or discovering new ones. This research is applicable in domains using AR Headsets with intuitive user controls for interactive applications, such as industry, manufacturing, aviation, education, entertainment, energy, and defense.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

增强现实（AR）允许将虚拟对象放置在真实的环境中。AR眼镜是下一波人机交互技术。这项技术将对教育、制造业、国防和装配工作等产生直接影响。理解人类使用手势、语音或两者（手势和语音）的组合与AR系统交互的作用将提供基础知识，以降低用户交互的复杂性。这项研究将通过支持代表性不足和非传统的研究生和开发新的跨学科课程在社会上产生影响。这项研究的成果将有助于创建改进的AR应用程序，使未来的劳动力培训等多个领域受益。通过该项目生成的数据集可以导致计算机科学家和认知/学习科学家之间的合作探索，使学习AR应用程序更容易访问，并包含更直观的交互。该项目将生成关于AR三维（3D）用户界面手势的初步基础研究，设计以自我为中心的手势，使用AR耳机进行语音交互，构建两个用于手势识别的标记数据集，并为AR耳机开发基于块的手势应用程序。该项目将通过以下方式推进AR头显手势交互的最新技术水平：（1）启发研究：该项目将进行迄今为止最大规模的启发研究，大幅增加基础知识库，并改进启发方法和手势识别器。这项研究将需要创建一个新的应用程序，该应用程序将生成新的协作交互数据集，从而通过手势诱导获得关于AR 3D用户界面的用户手势的更好的基础知识;（2）数据集生成：诱导研究将生成标记和未标记的以自我为中心的数据集，这将使人机交互和计算机视觉研究人员能够探索新的识别算法;（3）AR头戴式设备中的多模态交互（手势和语音）交互：作为该项目的一部分进行的初步研究将允许使用AR头戴式设备进行复杂交互和协作任务的研究。这项研究将包括进行以自我为中心的手势研究（有/没有语音），以确定AR耳机应用程序的适当交互。该项目的研究活动将扩大我们对动作/手势映射之外的启发研究的理解和使用，以及AR耳机中的多模式交互（手势和语音），无论是验证人与人之间的交流还是发现新的交流。这项研究适用于使用AR耳机的领域，这些耳机具有直观的用户控制功能，可用于交互式应用，如工业、制造业、航空、教育、娱乐、能源和国防。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Using a 6 Degrees of Freedom Virtual Reality Input Device With An Augmented Reality Headset In A Collaborative Environment

在协作环境中使用 6 自由度虚拟现实输入设备和增强现实耳机

• DOI: 10.1109/vrw52623.2021.00045
• 发表时间: 2021
• 期刊: 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW
• 作者: Williams, Adam S.;Ortega, Francisco R.
• 通讯作者: Ortega, Francisco R.

Survey of Annotations in Extended Reality Systems

扩展现实系统中的注释调查

• DOI: 10.1109/tvcg.2023.3288869
• 发表时间: 2023
• 期刊: IEEE Transactions on Visualization and Computer Graphics
• 影响因子: 5.2
• 作者: Borhani, Zahra;Sharma, Prashast;Ortega, Francisco R.
• 通讯作者: Ortega, Francisco R.

Remote XR Studies: Exploring Three Key Challenges of Remote XR Experimentation

远程 XR 研究：探索远程 XR 实验的三个关键挑战

• DOI: 10.1145/3411763.3442472
• 发表时间: 2021
• 期刊: CHI '21: CHI Conference on Human Factors in Computing Systems
• 作者: Ratcliffe, Jack;Soave, Francesco;Hoover, Melynda;Ortega, Francisco Raul;Bryan-Kinns, Nick;Tokarchuk, Laurissa;Farkhatdinov, Ildar
• 通讯作者: Farkhatdinov, Ildar

Understanding Gesture and Speech Multimodal Interactions for Manipulation Tasks in Augmented Reality Using Unconstrained Elicitation

使用无约束启发了解增强现实中操作任务的手势和语音多模态交互

• DOI: 10.1145/3427330
• 发表时间: 2020
• 期刊: Proceedings of the ACM on Human-Computer Interaction
• 作者: Williams, Adam S.;Ortega, Francisco R.
• 通讯作者: Ortega, Francisco R.

The Cost of Production in Elicitation Studies and the Legacy Bias-Consensus Trade off

启发研究中的生产成本和遗留偏差与共识的权衡

• DOI: 10.3390/mti4040088
• 发表时间: 2020
• 期刊: Multimodal Technologies and Interaction
• 影响因子: 2.5
• 作者: Williams, Adam S.;Garcia, Jason;De Zayas, Fernando;Hernandez, Fidel;Sharp, Julia;Ortega, Francisco R.
• 通讯作者: Ortega, Francisco R.