Collaborative Research: HCC: Medium: HCI in Motion -- Using EEG, Eye Tracking, and Body Sensing for Attention-Aware Mobile Mixed Reality

合作研究：HCC：媒介：运动中的 HCI——使用 EEG、眼动追踪和身体感应实现注意力感知移动混合现实

• 批准号: 2211784
• 负责人: Tobias Hollerer
• 金额: $ 73.87万
• 依托单位: University of California-Santa Barbara
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-09-01 至 2025-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2211784&HistoricalAwards=false
• 关键词: Collaborative; Research; HCC; Medium; HCI

Mobile, wireless, headsets for virtual and augmented reality, such as the Meta Quest 2 and Microsoft HoloLens-2, are becoming more widely used in many applications beyond video games, such as training, construction, and medicine. However, wearing these head-worn goggles while walking can make some people feel sick or distracted, which has even led to injury in some cases. This effect is similar to texting while walking, but potentially worse because a person's entire periphery can be filled with distracting media elements. While previous research has investigated these issues when users are standing still or seated, it is unclear how problems unfold and how they can be prevented while users are in motion. Specifically, this project will investigate how and why virtual and augmented reality headsets affect attention and feelings of sickness. First, this work will record data, such as heart rate, brain waves, and the direction users are turning their eyes to, while they are wearing virtual and augmented reality headsets and walking. Secondly, this project will develop ways to reduce sickness and distraction while walking with virtual and augmented reality headsets. This work will improve the safety of mobile virtual and augmented reality headsets, products that virtually all big technology companies today heavily invest in as possible companions or replacements to smartphones. This project will be introduced in courses and research mentorship projects at The University of Texas at San Antonio and the University of California at Santa Barbara, to advance research training of both undergraduate and graduate students. Considering that both universities and research teams have a history of supporting many underrepresented minority students, it is expected that the educational value of this project will be high, especially in terms of recruiting and mentoring women and underrepresented minority students.There is an increasing prevalence of mobile, immersive interfaces (e.g., mobile Virtual Reality(VR) / Augmented Reality (AR)) that may affect users' cognitive capacities and situational awareness, potentially leading to physical harm (e.g., impaired task performance, tripping over physical obstacles in VR, unsafe street crossings while seeing advertisements in AR). The landscape of human-computer interaction has expanded from fairly well standardized stationary office configurations to more varied mobile and immersive settings involving active body movements (mobile and situated computing, AR, mobile VR) and simulated first-person perspective changes and motion experiences (immersive computing). To make matters worse, compared to more standardized platforms such as desktop and laptop UIs, tablet and smartphone interfaces, individual differences among users have a much bigger usability impact in context-driven surround-focus usage scenarios found in mobile AR/VR. For example, motion sickness (i.e., cybersickness) in VR is known to inflict symptoms of widely varying severity, depending on the individual user. One serious consequence is that interaction designers have difficulties providing engaging general experiences that are universally usable by a wide variety of users. Despite the increasing prevalence of immersive technologies and their pitfalls, the precise cognitive and physiological mechanisms at play when 'computing in motion' are not well understood. This work is aimed at filling this knowledge gap. The specific objectives are: 1) to assess the cognitive effects of interacting with mobile AR/VR while users are walking, 2) to provide automated tools to effectively reduce the cognitive demand of mobile AR/VR, and 3) to make mobile AR/VR safer and more usable. Based on preliminary data, the central hypothesis is that through multi-modal sensing combined with machine learning approaches, mobile AR/VR applications can learn the characteristics of user behavior and provide real time adaptations that will reduce user error, increase ease of use, improve task performance, and reduce the impact of physical hazards. This work will improve the safety of mobile AR and mobile VR, paradigms that virtually all big technology companies today heavily invest in as a possible follow-up paradigm to the smartphone platform. Educational impact will occur through incorporation of research outcomes into undergraduate and graduate courses offered at The University of Texas at San Antonio and the University of California at Santa Barbara, and research training and mentorship opportunities for both undergraduate and graduate students. The courses include Machine Learning, Deep Learning for Visual Computing, Human-Computer Interaction, and Mobile Application Programming. Because our project integrates a topic of high social impact with cutting edge machine learning and human-computer interaction research along with proven successful mentorship strategies, the educational impacts of the project will be high, especially in terms of recruiting and mentoring women and underrepresented minority students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

用于虚拟和增强现实的移动的、无线的头戴式耳机，例如Meta Quest 2和Microsoft HoloLens-2，正越来越广泛地用于视频游戏以外的许多应用，例如培训、建筑和医疗。然而，走路时戴着这些头戴式护目镜会让一些人感到恶心或分心，在某些情况下甚至会导致受伤。这种影响类似于边走边发短信，但可能更糟，因为一个人的整个外围都可能充满了分散注意力的媒体元素。虽然以前的研究已经调查了用户站着不动或坐着时的这些问题，但目前还不清楚问题是如何展开的，以及如何在用户运动时预防这些问题。具体来说，该项目将研究虚拟和增强现实耳机如何以及为什么会影响注意力和疾病感。首先，这项工作将记录数据，如心率，脑电波，以及用户在佩戴虚拟和增强现实耳机和行走时将目光转向的方向。其次，该项目将开发减少疾病和分心的方法，同时使用虚拟和增强现实耳机行走。这项工作将提高移动的虚拟和增强现实头戴设备的安全性，这些产品如今几乎所有大型科技公司都在大力投资，作为智能手机的可能伴侣或替代品。该项目将被引入到德克萨斯大学圣安东尼奥分校和加州大学圣巴巴拉分校的课程和研究指导项目中，以推进本科生和研究生的研究培训。考虑到大学和研究团队都有支持许多代表性不足的少数民族学生的历史，预计该项目的教育价值将很高，特别是在招募和指导女性和代表性不足的少数民族学生方面。移动的虚拟现实（VR）/增强现实（AR）），可能会影响用户的认知能力和情景意识，可能导致身体伤害（例如，任务性能受损，在VR中被物理障碍物绊倒，在AR中看到广告时不安全的街道交叉口）。人机交互的前景已经从相当标准化的固定办公室配置扩展到更多样化的移动的和沉浸式设置，包括主动身体运动（移动的和情境计算、AR、移动的VR）和模拟的第一人称视角变化和运动体验（沉浸式计算）。更糟糕的是，与台式机和笔记本电脑UI、平板电脑和智能手机界面等更标准化的平台相比，用户之间的个体差异在移动的AR/VR中的上下文驱动的环绕聚焦使用场景中具有更大的可用性影响。例如，运动病（即，已知VR中的晕电脑病（cybersickness）会根据个人用户的不同而造成严重程度不同的症状。一个严重的后果是，交互设计师很难提供广泛的用户普遍可用的吸引人的一般体验。尽管沉浸式技术及其陷阱越来越普遍，但当“运动中的计算”时，精确的认知和生理机制还没有得到很好的理解。这项工作旨在填补这一知识空白。具体目标是：1）评估用户在行走时与移动的AR/VR交互的认知效果，2）提供自动化工具，有效降低移动的AR/VR的认知需求，3）让移动的AR/VR更安全、更可用。基于初步数据，中心假设是通过多模态传感与机器学习方法相结合，移动的AR/VR应用可以学习用户行为的特征，并提供真实的时间适应，从而减少用户错误，提高易用性，提高任务性能，并减少物理危害的影响。这项工作将提高移动的AR和移动的VR的安全性，这些模式是当今几乎所有大型科技公司都大力投资的，作为智能手机平台的可能后续模式。教育的影响将通过研究成果纳入本科和研究生课程在德克萨斯大学圣安东尼奥和加州大学圣巴巴拉提供，并为本科生和研究生的研究培训和导师的机会。课程包括机器学习，视觉计算深度学习，人机交互和移动的应用程序编程。由于我们的项目将具有高度社会影响力的主题与尖端机器学习和人机交互研究沿着，以及经过验证的成功导师策略相结合，因此该项目的教育影响将很高，该奖项反映了NSF的法定使命，并被认为是值得通过使用基金会的知识产权评估来支持的。优点和更广泛的影响审查标准。

项目成果

OCTOPUS: Open-vocabulary Content Tracking and Object Placement Using Semantic Understanding in Mixed Reality

• DOI: 10.1109/ismar-adjunct60411.2023.00124
• 发表时间: 2023-10
• 期刊: 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)
• 作者: Luke Yoffe;Aditya Sharma;Tobias Höllerer

Dynamic Theater: Location-Based Immersive Dance Theater, Investigating User Guidance and Experience

动态剧场：基于位置的沉浸式舞蹈剧场，研究用户引导和体验

• DOI: 10.1145/3611659.3615705
• 发表时间: 2023
• 期刊: VRST '23: Proceedings of the 29th ACM Symposium on Virtual Reality Software and Technology
• 作者: Kim, You-Jin;Lu, Joshua;Höllerer, Tobias
• 通讯作者: Höllerer, Tobias

Layerable Apps: Comparing Concurrent and Exclusive Display of Augmented Reality Applications

可分层应用程序：比较增强现实应用程序的并发和独占显示

• DOI: 10.1109/ismar55827.2022.00104
• 发表时间: 2022
• 期刊: Proceedings International Symposium on Mixed and Augmented Reality ISMAR
• 作者: Huynh, Brandon;Wysopal, Abby;Ross, Vivian;Orlosky, Jason;Hollerer, Tobias
• 通讯作者: Hollerer, Tobias

The Impact of Navigation Aids on Search Performance and Object Recall in Wide-Area Augmented Reality

导航辅助对广域增强现实中的搜索性能和对象回忆的影响

• DOI: 10.1145/3544548.3581413
• 发表时间: 2023
• 期刊: CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
• 作者: Kumaran, Radha;Kim, You-Jin;Milner, Anne E;Bullock, Tom;Giesbrecht, Barry;Höllerer, Tobias
• 通讯作者: Höllerer, Tobias

Benefits of Synthetically Pre-trained Depth-Prediction Networks for Indoor/Outdoor Image Classification

• DOI: 10.1109/wacvw58289.2023.00040
• 发表时间: 2023-01
• 期刊: 2023 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW)
• 作者: Ke Lin;Irene Cho;Ameya S. Walimbe;Bryan A. Zamora;Alex Rich;Sirius Z. Zhang;Tobias Höllerer