Expanding articulatory information from ultrasound imaging of speech using MRI-based image simulations and audio measurements

项目总结 超声成像提供了可用于治疗语音障碍的发音反馈， 这影响了5%的儿童，并导致成年后社会健康和就业方面的长期缺陷。 然而，超声成像对于临床医生和个人来说可能很难解释，从而限制了 理解发音数据和超声生物反馈治疗的语音结果。一个可能的来源 难点是超声图像中缺少发音信息，如舌尖和参考信息 声道结构(例如，腭部)由于空气的原因，无法用超声波始终如一地成像。 超声波遗漏的大部分信息可以在磁共振成像中得到确认 (核磁共振)，因为核磁共振成像整个声道。将超声图像和核磁共振图像进行比较将会有所改进 通过确认超声图像的某些特征(例如， 舌尖模糊、双边缘伪影)由舌形的特征产生；同样，模型可以 接受训练，从超声图像中预测核磁共振显示的关节信息。然而，发音的 可变性阻止了这些图像之间的直接比较。避免变化性的一种新方法是 模拟超声波在从MRI分割的组织中的传播。深度学习的最新进展 还展示了解决从声学数据预测发音的反问题的能力。 因此，为了满足提高超声图像解释的需要，本方案的目标是使用 用于表征和预测关节信息的模拟超声图像和神经网络模型 在2D矢状面超声图像中缺失。这些模型将接受关于核磁共振和音频数据的培训。 我们将通过开发高效的超声模拟来表征缺失的发音信息 从MRI组织分割得到的图像。将被检验的一个假设是使用较低的 超声图像中双边缘伪影的边缘作为舌面。要测试这一指导方针以获得更大的 数据范围(包括紊乱的儿童说话者和不同的模拟探头旋转)，双边边缘 伪像将与用于生成模拟图像的组织图进行比较。另一个比较将是 估计/r/舌形中通常缺失的舌尖数量。然后我们将开发一种深度学习 该模型根据来自MRI的信息进行训练，以预测中矢状声道形状(包括舌尖和 舌部)从超声波和音频输入的舌形。有了这些目标，我们将为 用于语音的超声成像并提供在扩展发音信息方面具有未来应用的工具， 例如，在超声生物反馈治疗中使用更完整的声道信息的测试结果。 该奖学金的培训将在辛辛那提大学进行，并有机会在下午2点参观实验室。 额外的机构。拟议的计划提供了来自一系列调查人员的培训，主题包括 超声成像和在语音研究中的应用，培养我未来目标所需的技能。

PROJECT SUMMARY Ultrasound imaging provides articulatory feedback useful for remediating speech sound disorders, which affect 5% of children and cause long-term deficits in social health and employment in adulthood. However, ultrasound imaging can be difficult to interpret for clinicians and individuals, limiting the understanding of articulatory data and ultrasound biofeedback therapy speech outcomes. A likely source of difficulty is the articulatory information missing from ultrasound images, such as the tongue tip and reference vocal tract structures (e.g., palate) that cannot be consistently imaged with ultrasound due to air. Much of this missing information from ultrasound can be ascertained in magnetic resonance imaging (MRI) because MRI images the entire vocal tract. Comparing ultrasound images and MRI will improve interpretation of ultrasound images by confirming that certain characteristics of ultrasound images (e.g., obscured tongue tip, double edge artifacts) occur from characteristics of tongue shapes; as well, models can be trained to predict from ultrasound images the articulatory information shown in MRI. However, articulatory variability prevents direct comparison between these images. A novel approach to avoid variability is to simulate ultrasound wave propagation in tissue segmented from MRI. Recent advancements in deep learning have also demonstrated ability to address the inverse problem of predicting articulation from acoustic data. Thus, to meet the needs of improving ultrasound image interpretation, the goal for this proposal is to use simulated ultrasound images and neural network models to characterize and predict articulatory information missing from 2D midsagittal ultrasound images. These models will be trained on MRI and audio data. We will characterize missing articulatory information by developing efficient simulation of ultrasound images from MRI tissue segmentation. One hypothesis that will be tested is the guideline for using the lower edge of double edge artifacts in ultrasound images as the tongue surface. To test this guideline for a greater range of data (including disordered child speakers and different simulated probe rotations), double edge artifacts will be compared with tissue maps used to generate the simulated images. Another comparison will estimate the amount of tongue tip typically missing in /r/ tongue shapes. We will then develop a deep learning model that trains on information from MRI to predict midsagittal vocal tract shapes (including the tongue tip and palate) from the inputs of tongue contours from ultrasound and audio. With these aims, we will add insight to ultrasound imaging for speech and provide a tool with future applications in expanding articulatory information, e.g., testing outcomes of using more complete vocal tract information in ultrasound biofeedback therapy. Training for this fellowship will occur at the University of Cincinnati, with opportunities to visit labs at two additional institutions. The proposed plan provides training from a range of investigators in topics such as ultrasound imaging and application to speech research, developing skills needed for my future goals.