Multimodal synchronous spatial-temporal acquisition of facial features and tongue for medical applications

用于医疗应用的面部特征和舌头的多模态同步时空采集

• 批准号: RGPIN-2014-05884
• 负责人: Popa, Tiberiu
• 金额: $ 1.82万
• 依托单位: Concordia University
• 依托单位国家: 加拿大
• 项目类别: Discovery Grants Program - Individual
• 财政年份: 2017
• 资助国家: 加拿大
• 起止时间: 2017-01-01 至 2018-12-31
• 项目状态: 已结题
• 来源: https://www.nserc-crsng.gc.ca/ase-oro/Details-Detailles_eng.asp?id=634642
• 关键词: Multimodal; synchronous; spatial; temporal; acquisition

High quality geometry acquisition of time varying surfaces, also known as spatio-temporal geometry reconstruction or 4D surface reconstruction, is a fundamental problem in the fields of computer graphics and computer vision with a very wide range of applications in several medical fields, engineering, tele-presence, entertainment, fashion, and e-commerce. One general goal of my research that I have pursued during my PhD and my post-doctoral studies is to develop fundamental spatio-temporal acquisition methods and integrate them seamlessly in various application domains such as garment capture, face performance capture, free viewpoint video, tele-presence, gaze correction to name a few. A key ingredient to these acquisition methodologies is the combination of multiple sensors to achieve high fidelity reconstruction results. Depending on the application domain, more sensor modalities can be aggregated together: video cameras, hybrid depth+color cameras, temperature cameras, CT-scan, ultra-sound data, MRI data etc. These new data types can provide additional complementary information that can help the reconstruction process. Such multi-modal acquisition systems have received to date relatively little examination from the research community, but they have a great potential particularly in the medical field. During my tenure at ETH Zurich we developed several such multi-modal acquisition methods with applications in real-time performance capture for tele-presence applications and multi-modal acquisition from 3D facial scans and CT-scans with applications in surgery assessment and evaluation. This proposal builds up on this work and following my expertise in multi-modal acquisition systems we will develop new methods that fuse the data from new sensor modalities for a variety of medical applications. There are many synergies and research questions on spatio-temporal acquisition with applications in speech and language therapy, assessment and evaluation, therapy in children with autism spectrum disorder, biofeedback and neurofeedback therapy of facial paralysis and facial kinesis, therapy and computer aided lip-reading for hearing impaired to name a few. For instance, fusing together ultrasound data and video data for real-time synchronous acquisition of tongue and facial performance can lead to better understanding of speech and language disorders and can provide new therapy tools. Combining electromyography (EMG) sensing technology that can measure electrical activity in facial muscles with spatio-temporal 3D acquisition of faces can lead to a better understanding of facial paralysis and related disorders as well as more effective 3D-assessment, evaluation, documentation, training program and therapy progress protocol. Accurate 3D lip motion acquisition can be used to document the lip movements to get optimal exploit of optical speech sound characteristics in speech and language therapies.The proposed research focuses on these three important problems: (a) Fuse ultrasound imaging with spatio-temporal 3D reconstruction of faces for synchronous simultaneous reconstruction of the tongue and face performance during speech (b) Fuse EMG data with with spatio-temporal 3D reconstruction of faces for a better understanding of the correlation between facial muscle and facial expressions and (c) Practical lip movement acquisition during speech using inexpensive consumer level hardware.

时变表面的高质量几何采集，也称为时空几何重建或4D表面重建，是计算机图形学和计算机视觉领域的一个基本问题，在多个医学领域、工程、远程呈现、娱乐、时尚和电子商务领域有着广泛的应用。我在博士和博士后研究期间追求的研究总体目标是开发基本的时空采集方法，并将其无缝集成到各种应用领域，例如服装捕捉、面部表演捕捉、自由视点视频、远程呈现、凝视校正等等。这些采集方法的关键要素是组合多个传感器以实现高保真度重建结果。根据应用领域，可以将更多传感器模式聚合在一起：摄像机、混合深度+彩色摄像机、温度摄像机、CT 扫描、超声数据、MRI 数据等。这些新数据类型可以提供额外的补充信息，帮助重建过程。迄今为止，这种多模式采集系统受到研究界的检验相对较少，但它们具有巨大的潜力，特别是在医学领域。在我在苏黎世联邦理工学院任职期间，我们开发了几种此类多模态采集方法，应用于远程呈现应用的实时性能捕获，以及从 3D 面部扫描和 CT 扫描进行多模态采集，应用于手术评估和评估。该提案建立在这项工作的基础上，并根据我在多模式采集系统方面的专业知识，我们将开发新方法，融合来自各种医疗应用的新传感器模式的数据。时空习得在言语和语言治疗、评估和评估、自闭症谱系障碍儿童治疗、面瘫和面部运动的生物反馈和神经反馈治疗、听力障碍治疗和计算机辅助唇读等方面的应用存在许多协同作用和研究问题。例如，将超声数据和视频数据融合在一起以实时同步采集舌头和面部表现可以更好地理解言语和语言障碍，并可以提供新的治疗工具。将可测量面部肌肉电活动的肌电图 (EMG) 传感技术与面部时空 3D 采集相结合，可以更好地了解面瘫和相关疾病，并提供更有效的 3D 评估、评价、记录、培训计划和治疗进展方案。准确的 3D 嘴唇运动采集可用于记录嘴唇运动，以便在言语和语言治疗中获得光学语音特征的最佳利用。拟议的研究重点关注以下三个重要问题：（a）将超声成像与面部时空 3D 重建相融合，以同步重建言语期间的舌头和面部表现（b）将 EMG 数据与 面部时空 3D 重建，以便更好地理解面部肌肉和面部表情之间的相关性；(c) 使用廉价的消费级硬件在语音过程中获取实用的嘴唇运动。