Collaborative Research: CCSS: Continuous Facial Sensing and 3D Reconstruction via Single-ear Wearable Biosensors

人脸标志点跟踪和三维重建是计算机视觉、图形学和机器学习交叉领域的热门研究领域。尽管有无数的应用，如人机交互，面部表情分析和情感识别，现有的基于摄像头的解决方案需要用户被限制在一个特定的位置，并在任何时候都面对摄像头没有遮挡。这种高度受限的设置阻止了它们被部署在许多新兴的应用场景中，在这些场景中，用户可能会进行三维身体/头部运动。该项目旨在提供一种新形式的单耳生物传感系统，可以不引人注目地，连续地，可靠地感知整个面部和眼睛的运动，跟踪主要的面部标志，并通过跨模态迁移学习进一步渲染3D面部动画。该项目的研究成果将突破耳戴式生物传感的极限，实现目前尚不可行的丰富传感能力，如无摄像头面部地标跟踪和实时3D面部重建等。依托该项目研究的学习模型，项目团队正在构建两个代表性应用，即：用于移动的虚拟现实（VR）/增强现实（AR）的面部感测，以及使用重建的面部标志动态的语音增强。该项目将大大推进可穿戴和生物传感技术以及跨多种传感模式的迁移学习。该项目正在弥合人脸的解剖学和肌肉知识与电气和计算建模技术之间的差距，以开发用于感测基于面部的生理信号的分析模型，硬件和软件库。特别是，该项目团队正在构建一个低功耗低噪声电路，使用单耳生物传感器来感知整个面部肌肉活动。该团队还在开发一种压缩算法，仅在检测到面部变化时才激活传感和通信组件，这可以显着增加电池寿命并降低可穿戴系统的计算成本。此外，为了实现无摄像头的3D面部重建，该团队正在开发一种跨模态学习模型，该模型由视觉面部地标检测网络和生物信号网络组成，其中视觉模型中包含的知识可以在训练期间转移到生物信号域。为了进一步增强模型的鲁棒性，该团队正在整合第三种模式（即，惯性传感器）到跨模态学习模型，并探索域适应和持续学习技术。此外，该团队正在探索模型压缩和加速技术，以实现在现有头戴式设备（如VR/AR耳机）上的设备部署。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Facial landmark tracking and 3D reconstruction are popular and well-studied fields in the intersection of computer vision, graphics, and machine learning. Despite their countless applications such as human-computer interaction, facial expressions analysis, and emotion recognition, existing camera-based solutions require users to be confined to a particular location and face a camera at all times without occlusions. This highly constrained setting prevents them from being deployed in many emerging application scenarios, in which users are likely to engage in three-dimensional body/head movements. This project aims to provide a new form of single-ear biosensing system that can unobtrusively, continuously, and reliably sense the entire facial and eye movements, track major facial landmarks, and further render 3D facial animations via cross-modal transfer learning. The research outcome of this project will push the limits of ear-worn biosensing to enable rich sensing capabilities that are currently infeasible, such as camera-free facial landmark tracking, and real-time 3D facial reconstruction, etc. Relying on the learning model studied in this project, the project team is building two representative applications, i.e., facial sensing for mobile virtual reality (VR)/augmented reality (AR), and speech enhancement using the reconstructed facial landmark dynamics. The project will substantially advance the wearable and biosensing techniques as well as transfer learning across multiple sensing modalities.The project is bridging the gap between the anatomical and muscular knowledge of the human face and electrical and computational modeling techniques to develop analytical models, hardware, and software libraries for sensing face-based physiological signals. In particular, the project team is building a low-power low-noise circuit to sense the entire facial muscle activities using single-ear biosensors. The team is also developing a compressing algorithm that activates the sensing and communication components only when facial changes are detected, which can significantly increase the battery lifetime and reduce the computational cost of the wearable system. Moreover, to enable camera-free 3D facial reconstruction, the team is developing a cross-modal learning model that consists of a visual facial landmark detection network and a biosignal network, in which knowledge embodied in the vision model can be transferred to the biosignal domain during training. To further enhance the model’s robustness, the team is integrating the third modality (i.e., inertial sensors) into the cross-modal learning model and exploring domain adaptation and continual learning techniques. Additionally, the team is exploring model compression and acceleration techniques to enable the on-device deployment on existing head-worn devices such as VR/AR headsetsThis award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.