Collaborative Research: HCC: Medium: HCI in Motion -- Using EEG, Eye Tracking, and Body Sensing for Attention-Aware Mobile Mixed Reality

合作研究：HCC：媒介：运动中的 HCI——使用 EEG、眼动追踪和身体感应实现注意力感知移动混合现实

• 批准号: 2211785
• 负责人: John Quarles
• 金额: $ 45.71万
• 依托单位: University of Texas at San Antonio
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-09-01 至 2025-08-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2211785&HistoricalAwards=false
• 关键词: Collaborative; Research; HCC; Medium; HCI

Mobile, wireless, headsets for virtual and augmented reality, such as the Meta Quest 2 and Microsoft HoloLens-2, are becoming more widely used in many applications beyond video games, such as training, construction, and medicine. However, wearing these head-worn goggles while walking can make some people feel sick or distracted, which has even led to injury in some cases. This effect is similar to texting while walking, but potentially worse because a person's entire periphery can be filled with distracting media elements. While previous research has investigated these issues when users are standing still or seated, it is unclear how problems unfold and how they can be prevented while users are in motion. Specifically, this project will investigate how and why virtual and augmented reality headsets affect attention and feelings of sickness. First, this work will record data, such as heart rate, brain waves, and the direction users are turning their eyes to, while they are wearing virtual and augmented reality headsets and walking. Secondly, this project will develop ways to reduce sickness and distraction while walking with virtual and augmented reality headsets. This work will improve the safety of mobile virtual and augmented reality headsets, products that virtually all big technology companies today heavily invest in as possible companions or replacements to smartphones. This project will be introduced in courses and research mentorship projects at The University of Texas at San Antonio and the University of California at Santa Barbara, to advance research training of both undergraduate and graduate students. Considering that both universities and research teams have a history of supporting many underrepresented minority students, it is expected that the educational value of this project will be high, especially in terms of recruiting and mentoring women and underrepresented minority students.There is an increasing prevalence of mobile, immersive interfaces (e.g., mobile Virtual Reality(VR) / Augmented Reality (AR)) that may affect users' cognitive capacities and situational awareness, potentially leading to physical harm (e.g., impaired task performance, tripping over physical obstacles in VR, unsafe street crossings while seeing advertisements in AR). The landscape of human-computer interaction has expanded from fairly well standardized stationary office configurations to more varied mobile and immersive settings involving active body movements (mobile and situated computing, AR, mobile VR) and simulated first-person perspective changes and motion experiences (immersive computing). To make matters worse, compared to more standardized platforms such as desktop and laptop UIs, tablet and smartphone interfaces, individual differences among users have a much bigger usability impact in context-driven surround-focus usage scenarios found in mobile AR/VR. For example, motion sickness (i.e., cybersickness) in VR is known to inflict symptoms of widely varying severity, depending on the individual user. One serious consequence is that interaction designers have difficulties providing engaging general experiences that are universally usable by a wide variety of users. Despite the increasing prevalence of immersive technologies and their pitfalls, the precise cognitive and physiological mechanisms at play when 'computing in motion' are not well understood. This work is aimed at filling this knowledge gap. The specific objectives are: 1) to assess the cognitive effects of interacting with mobile AR/VR while users are walking, 2) to provide automated tools to effectively reduce the cognitive demand of mobile AR/VR, and 3) to make mobile AR/VR safer and more usable. Based on preliminary data, the central hypothesis is that through multi-modal sensing combined with machine learning approaches, mobile AR/VR applications can learn the characteristics of user behavior and provide real time adaptations that will reduce user error, increase ease of use, improve task performance, and reduce the impact of physical hazards. This work will improve the safety of mobile AR and mobile VR, paradigms that virtually all big technology companies today heavily invest in as a possible follow-up paradigm to the smartphone platform. Educational impact will occur through incorporation of research outcomes into undergraduate and graduate courses offered at The University of Texas at San Antonio and the University of California at Santa Barbara, and research training and mentorship opportunities for both undergraduate and graduate students. The courses include Machine Learning, Deep Learning for Visual Computing, Human-Computer Interaction, and Mobile Application Programming. Because our project integrates a topic of high social impact with cutting edge machine learning and human-computer interaction research along with proven successful mentorship strategies, the educational impacts of the project will be high, especially in terms of recruiting and mentoring women and underrepresented minority students.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

移动，无线，虚拟和增强现实的耳机，例如Meta Quest 2和Microsoft Hololens-2，在视频游戏（例如培训，建筑和医学）之外的许多应用程序中都广泛使用。但是，走路时戴着这些头饰的护目镜会使某些人感到不适或分心，在某些情况下，这甚至导致受伤。这种效果类似于走路时发短信，但可能会更糟，因为一个人的整个外围可以充满分心的媒体元素。尽管以前的研究已经在用户站立或坐下时研究了这些问题，但尚不清楚问题如何发展以及在使用用户时如何预防它们。具体而言，该项目将研究虚拟和增强现实耳机如何以及为什么会影响注意力和疾病的感觉。首先，这项工作将记录数据，例如心率，脑电波以及用户的方向，当时他们戴着虚拟和增强的现实耳机和步行。其次，该项目将开发减少疾病和分心的方法，同时使用虚拟和增强现实耳机行走。这项工作将提高移动虚拟和增强现实耳机的安全性，这些产品几乎所有大型技术公司如今都大量投资于可能的伴侣或更换智能手机。该项目将在得克萨斯大学圣安东尼奥大学和加利福尼亚大学圣塔芭芭拉分校的课程和研究指导项目中介绍，以推动对本科生和研究生的研究培训。考虑到大学和研究团队都有支持许多人代表性不足的少数族裔学生的历史，预计该项目的教育价值将很高，尤其是在招募和指导妇女和代表性不足的少数群体方面，越来越多。情境意识，可能导致身体伤害（例如，任务绩效受损，绊倒VR的身体障碍，不安全的街道交叉点，同时在AR中看到广告）。人类计算机互动的景观已从相当标准化的固定办公室配置扩展到涉及主动运动（移动和位置计算，AR，移动VR）以及模拟的第一人称视角变化和运动体验（沉浸式计算）的更多移动和沉浸式设置。更糟糕的是，与更标准化的平台（例如台式机和笔记本电脑，平板电脑和智能手机界面）相比，用户之间的个体差异在上下文驱动的环绕式用户使用方案中具有更大的可用性影响。例如，已知VR中的运动疾病（即Cyber​​sickness）会根据个人使用者而导致严重程度差异的症状。一个严重的后果是，互动设计师在提供广泛使用的一般体验方面遇到困难，这些经验普遍可供多种用户使用。尽管沉浸式技术及其陷阱的患病率越来越高，但当“运动中的计算机计算”时，确切的认知和生理机制尚未得到充分了解。这项工作旨在填补这一知识差距。具体目标是：1）评估用户步行时与移动AR/VR相互作用的认知效果，2）提供自动化工具以有效地减少移动AR/VR的认知需求，以及3）使移动AR/VR更安全，并且更可用。基于初步数据，中心假设是，通过多模式感应结合机器学习方法，移动AR/VR应用程序可以学习用户行为的特征并提供实时适应，从而减少用户的错误，提高使用易用性，提高任务性能并减少身体危害的影响。这项工作将提高移动AR和移动VR的安全性，这些范式几乎所有大型技术公司如今都大量投资于智能手机平台可能的后续范式。教育影响将通过将研究成果纳入得克萨斯大学圣安东尼奥大学和加利福尼亚大学圣塔芭芭拉分校提供的本科和研究生课程以及研究生和研究生的研究培训和指导机会。这些课程包括机器学习，用于视觉计算的深度学习，人为计算机的交互和移动应用程序编程。因为我们的项目将具有高度社会影响的主题与最先进的机器学习和人类计算机的互动研究结合在一起，并将成功的指导策略融合在一起，因此该项目的教育影响将很高，尤其是在招募和指导妇女和代表性较低的少数民族学生方面，这一奖项反映了NSF的法定任务，并通过评估范围来进行评估的范围，并反映了众多的范围。

项目成果

LiteVR: Interpretable and Lightweight Cybersickness Detection using Explainable AI

LiteVR：使用可解释的人工智能进行可解释的轻量级晕眩检测

• DOI: 10.1109/vr55154.2023.00076
• 发表时间: 2023
• 期刊: 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR
• 作者: Kundu, Ripan Kumar;Islam, Rifatul;Quarles, John;Hoque, Khaza Anuarul
• 通讯作者: Hoque, Khaza Anuarul

Standing Balance Improvement Using Vibrotactile Feedback in Virtual Reality

在虚拟现实中使用振动触觉反馈改善站立平衡

• DOI: 10.1145/3562939.3565638
• 发表时间: 2022
• 期刊: ACM VRST 2022
• 作者: Mahmud, M. Rasel;Stewart, Michael;Cordova, Alberto;Quarles, John
• 通讯作者: Quarles, John

You Make Me Sick! The Effect of Stairs on Presence, Cybersickness, and Perception of Embodied Conversational Agents

• DOI: 10.1109/vr55154.2023.00071
• 发表时间: 2023-03
• 期刊: 2023 IEEE Conference Virtual Reality and 3D User Interfaces (VR)
• 作者: Samuel Ang;Amanda Fernandez;Michael Rushforth;J. Quarles

Reduction of cybersickness in head mounted displays use: A systematic review and taxonomy of current strategies

• DOI: 10.3389/frvir.2023.1027552
• 发表时间: 2023-03
• 期刊: Applied Sciences
• 作者: Samuel Ang;J. Quarles

Auditory Feedback to Make Walking in Virtual Reality More Accessible

听觉反馈让虚拟现实中的行走变得更容易

• DOI: 10.1109/ismar55827.2022.00103
• 发表时间: 2022
• 期刊: IEEE ISMAR 2022
• 作者: Mahmud, M. Rasel;Stewart, Michael;Cordova, Alberto;Quarles, John
• 通讯作者: Quarles, John