CHS: Small: Robust Eye Tracking and Facial Context Sensing for Mobile Augmented Reality and Virtual Reality

虚拟现实和增强现实耳机可能很快就会成为一种主要的大众市场移动消费电子设备。然而，在使这些头显更便携、更节能，以及通过提高用户注意力和情绪状态的意识来实现更身临其境的用户体验方面存在挑战。这个项目有两个目标。第一个目标是通过探索新的眼动仪设计与机器学习方法相结合，以低功耗但稳健的方式增强系统跟踪眼睛的能力。第二个目标是通过结合面部肌肉活动和摄像头等多种传感方式，研究从可穿戴式耳机中推断面部表情和用户情感的方法。该项目还将通过讲习班对中学生和代表性不足的学生产生教育影响。该项目涉及几个领域的活动和创新。在眼动追踪方面，该项目将探索采用立体摄像头和机器学习方法的新硬件设计，以增强对面部和眼睛形状以及眼镜运动的稳健性。在面部表情传感方面，该团队将探索结合眼电图（EOG）和多个摄像头视图的多模态传感方法，以推断表情并降低功耗。在网络系统方面，该项目将着眼于利用眼动追踪和面部表情感应，使无线卸载渲染到附近的计算节点。所研究的技术可以显著降低AR和VR系统的功耗，增强其感知人类表情的能力，以实现更身临其境的体验，并通过从无线连接的边缘云实现预测性预取来降低计算复杂性。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Virtual and Augmented Reality headsets may soon become a major mass-market mobile consumer electronics device. However, there are challenges in making these headsets more portable and less power hungry, and in enabling a more immersive user experience by increasing awareness of the user's attention and emotional state. This project's goals are two-fold. The first goal is to enhance systems' ability to track the eye in a low-power yet robust manner by exploring new eye tracker designs combined with machine learning approaches. The second goal is to study methods to infer facial expressions and user emotion from wearable headsets by combining multiple sensing modalities including facial muscle activity and cameras. The project will also have educational impact on middle-school students and under-represented students through workshops. The project involves activities and innovations in several areas. On the eye tracking side, the project will explore new hardware designs with stereo cameras and machine learning approaches to enhance robustness to face and eye shapes as well as eyeglass movements. On the facial expression sensing side, the team will explore multimodal sensing methods that combine electrooculography (EOG) and multiple camera views to infer expressions as well as to reduce power consumption. On the networked systems side, the project will look at leveraging eye tracking and facial expression sensing to enable wireless offload of rendering to a nearby compute node. The techniques to be researched can significantly lower the power consumed by AR and VR systems, enhance their ability to sense human expressions to enable more immersive experience, and reduce computational complexity by enabling predictive pre-fetch from a wirelessly connected edge cloud.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.