Collaborative Research: CCSS: Continuous Facial Sensing and 3D Reconstruction via Single-ear Wearable Biosensors

人脸地标跟踪和三维重建是计算机视觉、图形学和机器学习交叉领域的研究热点。尽管有无数的应用，如人机交互、面部表情分析和情感识别，但现有的基于相机的解决方案要求用户被限制在特定位置，并且始终面对相机而不被遮挡。这种高度受限的设置阻碍了它们在许多新兴应用场景中的部署，在这些场景中，用户可能会参与三维身体/头部运动。该项目旨在提供一种新型的单耳生物传感系统，该系统可以不引人注意地、连续地、可靠地感知整个面部和眼睛的运动，跟踪主要的面部地标，并通过跨模式迁移学习进一步渲染3D面部动画。该项目的研究成果将突破耳佩式生物传感的极限，使目前无法实现的丰富的感知能力，如无摄像头面部地标跟踪，实时三维人脸重建等。依托本项目研究的学习模型，项目组正在构建两个具有代表性的应用，即移动虚拟现实(VR)/增强现实(AR)的面部感知，以及使用重建的面部地标动态进行语音增强。该项目将极大地推进可穿戴技术和生物传感技术，并在多种传感模式之间转移学习。该项目正在弥合人脸解剖和肌肉知识与电气和计算建模技术之间的差距，以开发用于传感基于面部的生理信号的分析模型、硬件和软件库。特别是，该项目团队正在建造一种低功率、低噪音的电路，使用单耳生物传感器来感知整个面部肌肉的活动。该团队还在开发一种压缩算法，只有在检测到面部变化时才会激活传感和通信组件，这可以显著延长电池寿命并降低可穿戴系统的计算成本。此外，为了实现无摄像头的3D人脸重建，该团队正在开发一种跨模式学习模型，该模型由视觉面部地标检测网络和生物信号网络组成，其中视觉模型中包含的知识可以在训练期间转移到生物信号域。为了进一步增强模型的稳健性，该团队正在将第三种模式(即惯性传感器)集成到跨模式学习模型中，并探索领域适应和持续学习技术。此外，该团队正在探索模型压缩和加速技术，以实现在现有头戴式设备(如VR/AR耳机)上的设备部署。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Facial landmark tracking and 3D reconstruction are popular and well-studied fields in the intersection of computer vision, graphics, and machine learning. Despite their countless applications such as human-computer interaction, facial expressions analysis, and emotion recognition, existing camera-based solutions require users to be confined to a particular location and face a camera at all times without occlusions. This highly constrained setting prevents them from being deployed in many emerging application scenarios, in which users are likely to engage in three-dimensional body/head movements. This project aims to provide a new form of single-ear biosensing system that can unobtrusively, continuously, and reliably sense the entire facial and eye movements, track major facial landmarks, and further render 3D facial animations via cross-modal transfer learning. The research outcome of this project will push the limits of ear-worn biosensing to enable rich sensing capabilities that are currently infeasible, such as camera-free facial landmark tracking, and real-time 3D facial reconstruction, etc. Relying on the learning model studied in this project, the project team is building two representative applications, i.e., facial sensing for mobile virtual reality (VR)/augmented reality (AR), and speech enhancement using the reconstructed facial landmark dynamics. The project will substantially advance the wearable and biosensing techniques as well as transfer learning across multiple sensing modalities.The project is bridging the gap between the anatomical and muscular knowledge of the human face and electrical and computational modeling techniques to develop analytical models, hardware, and software libraries for sensing face-based physiological signals. In particular, the project team is building a low-power low-noise circuit to sense the entire facial muscle activities using single-ear biosensors. The team is also developing a compressing algorithm that activates the sensing and communication components only when facial changes are detected, which can significantly increase the battery lifetime and reduce the computational cost of the wearable system. Moreover, to enable camera-free 3D facial reconstruction, the team is developing a cross-modal learning model that consists of a visual facial landmark detection network and a biosignal network, in which knowledge embodied in the vision model can be transferred to the biosignal domain during training. To further enhance the model’s robustness, the team is integrating the third modality (i.e., inertial sensors) into the cross-modal learning model and exploring domain adaptation and continual learning techniques. Additionally, the team is exploring model compression and acceleration techniques to enable the on-device deployment on existing head-worn devices such as VR/AR headsetsThis award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.