EAGER: Attention-Aware Mixed Reality Interfaces

EAGER：注意力感知混合现实界面

• 批准号: 1845587
• 负责人: Tobias Hollerer
• 金额: $ 24.5万
• 依托单位: University of California-Santa Barbara
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-09-01 至 2020-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1845587&HistoricalAwards=false
• 关键词: EAGER; Attention; Aware; Mixed; Reality

This project develops new technologies that measure and model people's state of attention and applies these methodologies to virtual reality (VR) and augmented reality (AR) language learning applications. For determining the degree of people's attention on central learning tasks presented in VR and AR, this project uses two sensor modalities that can index attention: electrical activity of the brain, as measured by electroencephalography (EEG) signals, and eye gaze behavior, as measured by eye trackers. Context recognition plays a key element in future VR and AR application scenarios, and users, as well as content providers, can benefit substantially from information about user attention states during information consumption. The project can inform the development of optimized VR and AR content, as well as individualized learning strategies. The project's motivating application is the optimization of language learning for users across the complete spectrum of ability. In the longer run, additional benefits include the creation of special tools for students with known attention deficits, as well as for increasing productivity and safety in various commercial and industrial applications. This research explores necessary novel technologies for attention-aware mixed reality (MR) interfaces. The project integrates signals from consumer-grade EEG and eye tracking devices, to determine if and how much the participant's attention is divided (i.e. distracted or multi-tasking) or focused, and to assist appropriately. In this attention-assisted paradigm, users are monitored by EEG and eye tracking devices while interacting with mixed reality user interfaces. Attention states are classified over time using both sensor modalities and can be spatially referenced in the user interface with eye tracking. Attention activity feedback can be reported in real time while the user is interacting with the interface or may be stored and later visualized for a more thorough analysis of attentional patterns. The project delivers technology demonstrations of attention-aware interfaces for foreign language vocabulary learning in VR (with a lookout on AR possibilities). Exploratory experiments will uncover the possibilities for characterizing human attention states through EEG and eye tracking data. The project opens new opportunities for advances in multi-modal interaction to contribute to MR interfaces and learning technologies.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

该项目开发了测量和模拟人们注意力状态的新技术，并将这些方法应用于虚拟现实（VR）和增强现实（AR）语言学习应用。为了确定人们在VR和AR中呈现的中心学习任务上的注意力程度，该项目使用了两种可以索引注意力的传感器模式：通过脑电图（EEG）信号测量的大脑电活动，以及通过眼动仪测量的眼睛注视行为。上下文识别在未来的VR和AR应用场景中扮演着关键的角色，在信息消费过程中，用户以及内容提供商都可以从用户的注意力状态信息中获益。该项目可以为优化VR和AR内容的开发以及个性化学习策略提供信息。这个项目的激励应用是为各种能力的用户优化语言学习。从长远来看，其他好处包括为已知注意力缺陷的学生创造特殊工具，以及提高各种商业和工业应用的生产力和安全性。本研究探讨了注意感知混合现实（MR）接口所需的新技术。该项目集成了来自消费级脑电图和眼动追踪设备的信号，以确定参与者的注意力是否分散（即分心或多任务处理）或集中的程度，并提供适当的帮助。在这种注意力辅助范例中，用户在与混合现实用户界面交互时，通过脑电图和眼动追踪设备进行监测。注意状态随时间分类使用传感器模态，并可以在用户界面中通过眼动追踪进行空间引用。当用户与界面交互时，注意力活动反馈可以实时报告，或者可以存储并稍后可视化，以便对注意力模式进行更彻底的分析。该项目提供了在VR中用于外语词汇学习的注意力感知界面的技术演示（同时关注AR的可能性）。探索性实验将揭示通过脑电图和眼动追踪数据表征人类注意力状态的可能性。该项目为多模态交互的进步提供了新的机会，为MR接口和学习技术做出了贡献。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Designing a Multitasking Interface for Object-aware AR applications

为对象感知 AR 应用程序设计多任务界面

• DOI: 10.1109/ismar-adjunct51615.2020.00025
• 发表时间: 2020
• 期刊: 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct
• 作者: Huynh, Brandon;Orlosky, Jason;Hollerer, Tobias
• 通讯作者: Hollerer, Tobias

ARbis Pictus: A Study of Vocabulary Learning with Augmented Reality

• DOI: 10.1109/tvcg.2018.2868568
• 发表时间: 2017-11
• 期刊: IEEE Transactions on Visualization and Computer Graphics
• 影响因子: 5.2
• 作者: Adam Ibrahim;Brandon Huynh;J. Downey;Tobias Höllerer;Dorothy M. Chun;J. O'Donovan

In-Situ Labeling for Augmented Reality Language Learning

• DOI: 10.1109/vr.2019.8798358
• 发表时间: 2019-03
• 期刊: 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)
• 作者: Brandon Huynh;J. Orlosky;Tobias Höllerer

Augmentation Strategies for Learning with Noisy Labels

• DOI: 10.1109/cvpr46437.2021.00793
• 发表时间: 2021-03
• 期刊: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
• 作者: Kento Nishi;Yi Ding;Alex Rich;Tobias Höllerer

Predicting Video Affect via Induced Affection in the Wild

• DOI: 10.1145/3382507.3418838
• 发表时间: 2020-10
• 期刊: Proceedings of the 2020 International Conference on Multimodal Interaction
• 作者: Yi Ding;Radha Kumaran;Tianjiao Yang;Tobias Höllerer