CHS: Small: Robust Eye Tracking and Facial Context Sensing for Mobile Augmented Reality and Virtual Reality

虚拟和增强现实耳机可能很快就会成为主要的大众市场移动消费电子设备。然而，让这些耳机变得更便携、更省电，以及通过提高对用户注意力和情绪状态的感知来实现更身临其境的用户体验，都存在挑战。这个项目的目标有两个。第一个目标是通过探索结合机器学习方法的新眼睛跟踪器设计，增强系统以低功耗但健壮的方式跟踪眼睛的能力。第二个目标是研究通过结合包括面部肌肉活动和摄像头在内的多种传感模式，从可穿戴耳机中推断面部表情和用户情绪的方法。该项目还将通过讲习班对中学生和代表性不足的学生产生教育影响。该项目涉及多个领域的活动和创新。在眼睛跟踪方面，该项目将探索具有立体摄像头和机器学习方法的新硬件设计，以增强对人脸和眼睛形状以及眼镜运动的稳健性。在面部表情传感方面，该团队将探索多模式传感方法，将电眼震(EOG)和多摄像头视角相结合，以推断表情并降低功耗。在联网系统方面，该项目将着眼于利用眼睛跟踪和面部表情传感来实现将渲染无线卸载到附近的计算节点。将要研究的技术可以显著降低AR和VR系统的功耗，增强它们感知人类表情的能力，以实现更身临其境的体验，并通过从无线连接的边缘云进行预测性预取来降低计算复杂性。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Virtual and Augmented Reality headsets may soon become a major mass-market mobile consumer electronics device. However, there are challenges in making these headsets more portable and less power hungry, and in enabling a more immersive user experience by increasing awareness of the user's attention and emotional state. This project's goals are two-fold. The first goal is to enhance systems' ability to track the eye in a low-power yet robust manner by exploring new eye tracker designs combined with machine learning approaches. The second goal is to study methods to infer facial expressions and user emotion from wearable headsets by combining multiple sensing modalities including facial muscle activity and cameras. The project will also have educational impact on middle-school students and under-represented students through workshops. The project involves activities and innovations in several areas. On the eye tracking side, the project will explore new hardware designs with stereo cameras and machine learning approaches to enhance robustness to face and eye shapes as well as eyeglass movements. On the facial expression sensing side, the team will explore multimodal sensing methods that combine electrooculography (EOG) and multiple camera views to infer expressions as well as to reduce power consumption. On the networked systems side, the project will look at leveraging eye tracking and facial expression sensing to enable wireless offload of rendering to a nearby compute node. The techniques to be researched can significantly lower the power consumed by AR and VR systems, enhance their ability to sense human expressions to enable more immersive experience, and reduce computational complexity by enabling predictive pre-fetch from a wirelessly connected edge cloud.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.