Multimodal synchronous spatial-temporal acquisition of facial features and tongue for medical applications

点击翻译按钮获取中文摘要

High quality geometry acquisition of time varying surfaces, also known as spatio-temporal geometry reconstruction or 4D surface reconstruction, is a fundamental problem in the fields of computer graphics and computer vision with a very wide range of applications in several medical fields, engineering, tele-presence, entertainment, fashion, and e-commerce. One general goal of my research that I have pursued during my PhD and my post-doctoral studies is to develop fundamental spatio-temporal acquisition methods and integrate them seamlessly in various application domains such as garment capture, face performance capture, free viewpoint video, tele-presence, gaze correction to name a few. A key ingredient to these acquisition methodologies is the combination of multiple sensors to achieve high fidelity reconstruction results. Depending on the application domain, more sensor modalities can be aggregated together: video cameras, hybrid depth+color cameras, temperature cameras, CT-scan, ultra-sound data, MRI data etc. These new data types can provide additional complementary information that can help the reconstruction process. Such multi-modal acquisition systems have received to date relatively little examination from the research community, but they have a great potential particularly in the medical field. During my tenure at ETH Zurich we developed several such multi-modal acquisition methods with applications in real-time performance capture for tele-presence applications and multi-modal acquisition from 3D facial scans and CT-scans with applications in surgery assessment and evaluation. This proposal builds up on this work and following my expertise in multi-modal acquisition systems we will develop new methods that fuse the data from new sensor modalities for a variety of medical applications. There are many synergies and research questions on spatio-temporal acquisition with applications in speech and language therapy, assessment and evaluation, therapy in children with autism spectrum disorder, biofeedback and neurofeedback therapy of facial paralysis and facial kinesis, therapy and computer aided lip-reading for hearing impaired to name a few. For instance, fusing together ultrasound data and video data for real-time synchronous acquisition of tongue and facial performance can lead to better understanding of speech and language disorders and can provide new therapy tools. Combining electromyography (EMG) sensing technology that can measure electrical activity in facial muscles with spatio-temporal 3D acquisition of faces can lead to a better understanding of facial paralysis and related disorders as well as more effective 3D-assessment, evaluation, documentation, training program and therapy progress protocol. Accurate 3D lip motion acquisition can be used to document the lip movements to get optimal exploit of optical speech sound characteristics in speech and language therapies. The proposed research focuses on these three important problems: (a) Fuse ultrasound imaging with spatio-temporal 3D reconstruction of faces for synchronous simultaneous reconstruction of the tongue and face performance during speech (b) Fuse EMG data with with spatio-temporal 3D reconstruction of faces for a better understanding of the correlation between facial muscle and facial expressions and (c) Practical lip movement acquisition during speech using inexpensive consumer level hardware.