Multimodal synchronous spatial-temporal acquisition of facial features and tongue for medical applications

高质量的时变曲面几何获取，也称为时空几何重建或4D曲面重建，是计算机图形学和计算机视觉领域的一个基本问题，在医学、工程、临场感、娱乐、时尚和电子商务等领域有着非常广泛的应用。在我的博士和博士后研究期间，我所从事的研究的一个总体目标是开发基本的时空采集方法，并将它们无缝地集成到各种应用领域，如服装捕捉、面部行为捕捉、自由视点视频、远程呈现、凝视校正等。这些采集方法的一个关键因素是组合多个传感器以实现高保真重建结果。根据应用领域的不同，可以将更多传感器类型聚合在一起：视频摄像机、混合深度+彩色摄像机、温度摄像机、CT扫描、超声数据、MRI数据等。这些新的数据类型可以提供额外的补充信息，帮助重建过程。到目前为止，这种多模式采集系统得到的研究相对较少，但它们有很大的潜力，特别是在医学领域。在我在苏黎世工业大学任职期间，我们开发了几种这样的多模式采集方法，应用于远程呈现应用的实时性能捕获，以及3D面部扫描和CT扫描的多模式采集，应用于手术评估和评估。这项建议建立在这项工作的基础上，根据我在多模式采集系统方面的专业知识，我们将开发新的方法，融合来自新传感器模式的数据，用于各种医疗应用。时空习得在言语和语言治疗、评估和评估、自闭症谱系障碍儿童的治疗、面瘫和面部运动的生物反馈和神经反馈治疗、听力障碍的治疗和计算机辅助唇读等方面有许多协同作用和研究问题。例如，将超声数据和视频数据融合在一起，实时同步获取舌头和面部表现，可以更好地理解言语和语言障碍，并可以提供新的治疗工具。将可以测量面部肌肉电活动的肌电(EMG)传感技术与面部的时空3D采集相结合，可以更好地了解面部瘫痪和相关疾病，以及更有效的3D评估、评估、记录、培训计划和治疗进展方案。准确的3D唇动采集可以用来记录唇部运动，从而在言语和语言治疗中最大限度地利用光学语音特征。这项研究主要集中在以下三个重要问题上：(A)将超声成像与人脸的时空3D重建相融合，以同步重建舌头和语音过程中的人脸表现；(B)将EMG数据与人脸的时空3D重建相融合，以更好地了解面部肌肉和面部表情之间的相关性；以及(C)使用廉价的消费者级硬件，在语音过程中获取实际的嘴唇运动。

High quality geometry acquisition of time varying surfaces, also known as spatio-temporal geometry reconstruction or 4D surface reconstruction, is a fundamental problem in the fields of computer graphics and computer vision with a very wide range of applications in several medical fields, engineering, tele-presence, entertainment, fashion, and e-commerce. One general goal of my research that I have pursued during my PhD and my post-doctoral studies is to develop fundamental spatio-temporal acquisition methods and integrate them seamlessly in various application domains such as garment capture, face performance capture, free viewpoint video, tele-presence, gaze correction to name a few. A key ingredient to these acquisition methodologies is the combination of multiple sensors to achieve high fidelity reconstruction results. Depending on the application domain, more sensor modalities can be aggregated together: video cameras, hybrid depth+color cameras, temperature cameras, CT-scan, ultra-sound data, MRI data etc. These new data types can provide additional complementary information that can help the reconstruction process. Such multi-modal acquisition systems have received to date relatively little examination from the research community, but they have a great potential particularly in the medical field. During my tenure at ETH Zurich we developed several such multi-modal acquisition methods with applications in real-time performance capture for tele-presence applications and multi-modal acquisition from 3D facial scans and CT-scans with applications in surgery assessment and evaluation. This proposal builds up on this work and following my expertise in multi-modal acquisition systems we will develop new methods that fuse the data from new sensor modalities for a variety of medical applications. There are many synergies and research questions on spatio-temporal acquisition with applications in speech and language therapy, assessment and evaluation, therapy in children with autism spectrum disorder, biofeedback and neurofeedback therapy of facial paralysis and facial kinesis, therapy and computer aided lip-reading for hearing impaired to name a few. For instance, fusing together ultrasound data and video data for real-time synchronous acquisition of tongue and facial performance can lead to better understanding of speech and language disorders and can provide new therapy tools. Combining electromyography (EMG) sensing technology that can measure electrical activity in facial muscles with spatio-temporal 3D acquisition of faces can lead to a better understanding of facial paralysis and related disorders as well as more effective 3D-assessment, evaluation, documentation, training program and therapy progress protocol. Accurate 3D lip motion acquisition can be used to document the lip movements to get optimal exploit of optical speech sound characteristics in speech and language therapies. The proposed research focuses on these three important problems: (a) Fuse ultrasound imaging with spatio-temporal 3D reconstruction of faces for synchronous simultaneous reconstruction of the tongue and face performance during speech (b) Fuse EMG data with with spatio-temporal 3D reconstruction of faces for a better understanding of the correlation between facial muscle and facial expressions and (c) Practical lip movement acquisition during speech using inexpensive consumer level hardware.