Collaborative Research: CCSS: Continuous Facial Sensing and 3D Reconstruction via Single-ear Wearable Biosensors

面部地标跟踪和3D重建是计算机视觉、图形学和机器学习交叉领域的热门研究领域。尽管有无数的应用，如人机交互、面部表情分析和情感识别，现有的基于摄像头的解决方案要求用户被限制在一个特定的位置，并且在任何时候都没有遮挡地面对摄像头。这种高度受限的设置使它们无法部署在许多新兴的应用场景中，在这些场景中，用户可能会进行三维身体/头部运动。本项目旨在提供一种新型的单耳生物传感系统，该系统可以不显眼地、连续地、可靠地感知整个面部和眼睛的运动，跟踪主要的面部标志，并通过跨模态迁移学习进一步渲染3D面部动画。本项目的研究成果将突破耳戴式生物传感的极限，实现目前无法实现的丰富传感功能，如无摄像头面部地标跟踪、实时3D面部重建等。依托本项目研究的学习模型，项目团队正在构建两个具有代表性的应用，即移动虚拟现实(VR)/增强现实（AR）的面部感知，以及利用重构的面部地标动态进行语音增强。该项目将大大推进可穿戴和生物传感技术，以及跨多种传感模式的迁移学习。该项目弥合了人脸解剖和肌肉知识与电子和计算建模技术之间的差距，以开发分析模型、硬件和软件库，用于感知基于面部的生理信号。特别是，项目组正在构建一个低功耗、低噪音的电路，利用单耳生物传感器来感知整个面部肌肉的活动。该团队还在开发一种压缩算法，该算法仅在检测到面部变化时才激活传感和通信组件，这可以显着增加电池寿命并降低可穿戴系统的计算成本。此外，为了实现无摄像头的3D面部重建，该团队正在开发一种跨模式学习模型，该模型由视觉面部地标检测网络和生物信号网络组成，其中视觉模型中包含的知识可以在训练过程中转移到生物信号领域。为了进一步增强模型的鲁棒性，该团队正在将第三种模态（即惯性传感器）集成到跨模态学习模型中，并探索领域自适应和持续学习技术。此外，该团队正在探索模型压缩和加速技术，以便在现有的头戴式设备（如VR/AR头戴式设备）上进行设备上部署。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Facial landmark tracking and 3D reconstruction are popular and well-studied fields in the intersection of computer vision, graphics, and machine learning. Despite their countless applications such as human-computer interaction, facial expressions analysis, and emotion recognition, existing camera-based solutions require users to be confined to a particular location and face a camera at all times without occlusions. This highly constrained setting prevents them from being deployed in many emerging application scenarios, in which users are likely to engage in three-dimensional body/head movements. This project aims to provide a new form of single-ear biosensing system that can unobtrusively, continuously, and reliably sense the entire facial and eye movements, track major facial landmarks, and further render 3D facial animations via cross-modal transfer learning. The research outcome of this project will push the limits of ear-worn biosensing to enable rich sensing capabilities that are currently infeasible, such as camera-free facial landmark tracking, and real-time 3D facial reconstruction, etc. Relying on the learning model studied in this project, the project team is building two representative applications, i.e., facial sensing for mobile virtual reality (VR)/augmented reality (AR), and speech enhancement using the reconstructed facial landmark dynamics. The project will substantially advance the wearable and biosensing techniques as well as transfer learning across multiple sensing modalities.The project is bridging the gap between the anatomical and muscular knowledge of the human face and electrical and computational modeling techniques to develop analytical models, hardware, and software libraries for sensing face-based physiological signals. In particular, the project team is building a low-power low-noise circuit to sense the entire facial muscle activities using single-ear biosensors. The team is also developing a compressing algorithm that activates the sensing and communication components only when facial changes are detected, which can significantly increase the battery lifetime and reduce the computational cost of the wearable system. Moreover, to enable camera-free 3D facial reconstruction, the team is developing a cross-modal learning model that consists of a visual facial landmark detection network and a biosignal network, in which knowledge embodied in the vision model can be transferred to the biosignal domain during training. To further enhance the model’s robustness, the team is integrating the third modality (i.e., inertial sensors) into the cross-modal learning model and exploring domain adaptation and continual learning techniques. Additionally, the team is exploring model compression and acceleration techniques to enable the on-device deployment on existing head-worn devices such as VR/AR headsetsThis award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.