Multimodal synchronous spatial-temporal acquisition of facial features and tongue for medical applications

时变表面的高质量几何获取，也称为时空几何重建或4D表面重建，是计算机图形学和计算机视觉领域中的基本问题，在若干医学领域、工程、远程呈现、娱乐、时尚和电子商务中具有非常广泛的应用。我在博士和博士后研究期间所追求的研究的一个总体目标是开发基本的时空采集方法，并将其无缝集成到各种应用领域，例如服装捕获，面部表演捕获，自由视点视频，远程呈现，凝视校正等。这些采集方法的一个关键因素是多个传感器的组合，以实现高保真度的重建结果。** 根据应用领域的不同，可以将更多传感器模态聚合在一起：视频摄像机、混合深度+彩色摄像机、温度摄像机、CT扫描、超声数据、MRI数据等。这些新的数据类型可以提供额外的补充信息，有助于重建过程。这样的多模态采集系统已经收到了迄今为止相对较少的检查，从研究界，但他们有一个很大的潜力，特别是在医疗领域。在我任职于苏黎世联邦理工学院期间，我们开发了几种这样的多模态采集方法，应用于远程呈现应用的实时性能捕获，以及来自3D面部扫描和CT扫描的多模态采集，应用于手术评估和评价。** 本提案建立在这项工作的基础上，并根据我在多模态采集系统方面的专业知识，我们将开发新的方法，将来自新传感器模态的数据融合到各种医疗应用中。有许多协同作用和研究问题的时空收购与应用在语音和语言治疗，评估和评价，自闭症谱系障碍儿童的治疗，生物反馈和神经反馈治疗面瘫和面部运动，治疗和计算机辅助唇读听力受损的仅举几例。例如，将超声数据和视频数据融合在一起，实时同步采集舌头和面部表现，可以更好地理解言语和语言障碍，并可以提供新的治疗工具。结合肌电图（EMG）传感技术，可以测量面部肌肉的电活动与时空3D面部采集，可以更好地了解面瘫和相关疾病，以及更有效的3D评估，评估，记录，培训计划和治疗进展协议。精确的3D嘴唇运动采集可用于记录嘴唇运动，以在语音和语言治疗中最佳地利用光学语音声音特征。**拟议的研究重点是这三个重要问题：（a）具有面部的时空3D重建的超声波成像，用于在讲话期间同步重建舌头和面部表现（B）具有面部的时空3D重建的超声波EMG数据，用于更好地理解面部肌肉和面部表情之间的相关性，以及（c）使用廉价的消费者级硬件在讲话过程中获得实际的嘴唇运动。

High quality geometry acquisition of time varying surfaces, also known as spatio-temporal geometry reconstruction or 4D surface reconstruction, is a fundamental problem in the fields of computer graphics and computer vision with a very wide range of applications in several medical fields, engineering, tele-presence, entertainment, fashion, and e-commerce. One general goal of my research that I have pursued during my PhD and my post-doctoral studies is to develop fundamental spatio-temporal acquisition methods and integrate them seamlessly in various application domains such as garment capture, face performance capture, free viewpoint video, tele-presence, gaze correction to name a few. A key ingredient to these acquisition methodologies is the combination of multiple sensors to achieve high fidelity reconstruction results. **Depending on the application domain, more sensor modalities can be aggregated together: video cameras, hybrid depth+color cameras, temperature cameras, CT-scan, ultra-sound data, MRI data etc. These new data types can provide additional complementary information that can help the reconstruction process. Such multi-modal acquisition systems have received to date relatively little examination from the research community, but they have a great potential particularly in the medical field. During my tenure at ETH Zurich we developed several such multi-modal acquisition methods with applications in real-time performance capture for tele-presence applications and multi-modal acquisition from 3D facial scans and CT-scans with applications in surgery assessment and evaluation. **This proposal builds up on this work and following my expertise in multi-modal acquisition systems we will develop new methods that fuse the data from new sensor modalities for a variety of medical applications. There are many synergies and research questions on spatio-temporal acquisition with applications in speech and language therapy, assessment and evaluation, therapy in children with autism spectrum disorder, biofeedback and neurofeedback therapy of facial paralysis and facial kinesis, therapy and computer aided lip-reading for hearing impaired to name a few. For instance, fusing together ultrasound data and video data for real-time synchronous acquisition of tongue and facial performance can lead to better understanding of speech and language disorders and can provide new therapy tools. Combining electromyography (EMG) sensing technology that can measure electrical activity in facial muscles with spatio-temporal 3D acquisition of faces can lead to a better understanding of facial paralysis and related disorders as well as more effective 3D-assessment, evaluation, documentation, training program and therapy progress protocol. Accurate 3D lip motion acquisition can be used to document the lip movements to get optimal exploit of optical speech sound characteristics in speech and language therapies.**The proposed research focuses on these three important problems: (a) Fuse ultrasound imaging with spatio-temporal 3D reconstruction of faces for synchronous simultaneous reconstruction of the tongue and face performance during speech (b) Fuse EMG data with with spatio-temporal 3D reconstruction of faces for a better understanding of the correlation between facial muscle and facial expressions and (c) Practical lip movement acquisition during speech using inexpensive consumer level hardware.