CHS: Small: Robust Eye Tracking and Facial Context Sensing for Mobile Augmented Reality and Virtual Reality

CHS：小型：用于移动增强现实和虚拟现实的强大眼动追踪和面部情境感知

• 批准号: 1815347
• 负责人: Deepak Ganesan
• 金额: $ 50万
• 依托单位: University of Massachusetts Amherst
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-09-01 至 2023-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1815347&HistoricalAwards=false
• 关键词: CHS; Small; Robust; Eye; Tracking

Virtual and Augmented Reality headsets may soon become a major mass-market mobile consumer electronics device. However, there are challenges in making these headsets more portable and less power hungry, and in enabling a more immersive user experience by increasing awareness of the user's attention and emotional state. This project's goals are two-fold. The first goal is to enhance systems' ability to track the eye in a low-power yet robust manner by exploring new eye tracker designs combined with machine learning approaches. The second goal is to study methods to infer facial expressions and user emotion from wearable headsets by combining multiple sensing modalities including facial muscle activity and cameras. The project will also have educational impact on middle-school students and under-represented students through workshops. The project involves activities and innovations in several areas. On the eye tracking side, the project will explore new hardware designs with stereo cameras and machine learning approaches to enhance robustness to face and eye shapes as well as eyeglass movements. On the facial expression sensing side, the team will explore multimodal sensing methods that combine electrooculography (EOG) and multiple camera views to infer expressions as well as to reduce power consumption. On the networked systems side, the project will look at leveraging eye tracking and facial expression sensing to enable wireless offload of rendering to a nearby compute node. The techniques to be researched can significantly lower the power consumed by AR and VR systems, enhance their ability to sense human expressions to enable more immersive experience, and reduce computational complexity by enabling predictive pre-fetch from a wirelessly connected edge cloud.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

虚拟和增强现实头戴式耳机可能很快成为一种主要的大众市场移动的消费电子设备。然而，在使这些头戴式耳机更便携和更少耗电方面存在挑战，并且在通过增加对用户的注意力和情绪状态的感知来实现更沉浸式的用户体验方面存在挑战。该项目的目标是双重的。第一个目标是通过探索结合机器学习方法的新眼动仪设计，增强系统以低功耗但鲁棒的方式跟踪眼睛的能力。第二个目标是研究通过结合包括面部肌肉活动和摄像头在内的多种传感方式来推断可穿戴耳机的面部表情和用户情绪的方法。该项目还将通过讲习班对中学生和代表性不足的学生产生教育影响。该项目涉及多个领域的活动和创新。在眼动追踪方面，该项目将探索采用立体摄像头和机器学习方法的新硬件设计，以增强对面部和眼睛形状以及面部运动的鲁棒性。在面部表情传感方面，该团队将探索多模态传感方法，结合联合收割机眼电图（EOG）和多个摄像头视图来推断表情并降低功耗。在网络系统方面，该项目将着眼于利用眼动跟踪和面部表情传感来实现渲染到附近计算节点的无线卸载。待研究的技术可以显著降低AR和VR系统的功耗，增强它们感知人类表情的能力，从而实现更身临其境的体验，并通过启用预测预处理来降低计算复杂度，该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查进行评估，被认为值得支持的搜索.

项目成果

Low-latency speculative inference on distributed multi-modal data streams

• DOI: 10.1145/3458864.3467884
• 发表时间: 2021-06
• 期刊: Proceedings of the 19th Annual International Conference on Mobile Systems, Applications, and Services
• 作者: Tianxing Li;Jin Huang;Erik Risinger;Deepak Ganesan

PhyMask: Robust Sensing of Brain Activity and Physiological Signals During Sleep with an All-textile Eye Mask

• DOI: 10.1145/3513023
• 发表时间: 2021-06
• 期刊: ACM Transactions on Computing for Healthcare (HEALTH)
• 作者: Soha Rostaminia;S. Z. Homayounfar;A. Kiaghadi;Trisha L. Andrew;Deepak Ganesan

CurtainNet: Enabling precise beamforming with a deformable antenna array on a fabric substrate

• DOI: 10.1145/3625687.3625787
• 发表时间: 2023-11
• 期刊: Proceedings of the 21st ACM Conference on Embedded Networked Sensor Systems
• 作者: Xingda Chen;Ankur Aditya;Zhenyu Lei;Deepak Ganesan

Re-thinking computation offload for efficient inference on IoT devices with duty-cycled radios

• DOI: 10.1145/3570361.3592514
• 发表时间: 2023-07
• 期刊: Proceedings of the 29th Annual International Conference on Mobile Computing and Networking
• 作者: Jin Huang;H. Guan;Deepak Ganesan

CLIO: enabling automatic compilation of deep learning pipelines across IoT and cloud

• DOI: 10.1145/3372224.3419215
• 发表时间: 2020-09
• 期刊: Proceedings of the 26th Annual International Conference on Mobile Computing and Networking
• 作者: Jin Huang;Colin Samplawski;Deepak Ganesan;Benjamin M. Marlin;H. Kwon