RI: Small: Modeling Idiosyncrasies of Speech for Automatic Spoken Language Processing

口语编码了音调和能量动态（韵律）以及不流畅（自我编辑）的重要信息，听者利用这些信息来理解说话者的意思和社会/情感背景。由于缺乏这些现象的适当模型，目前的语音处理系统很少利用这些信息。该项目通过关注意想不到的语音现象，假设这些事件通常携带最有价值的信息，并通过处理来自各种社会背景的语音，解决了建模的局限性。这项工作的应用范围从读写能力评估到改进人机交互。此外，了解不同的不流畅在非临床言语中的交际作用将有助于更准确的临床诊断。教育方面的目标是通过短期课程、学生TED演讲，将研究方法广泛地展示给不同学术水平的学生群体，并与华盛顿大学的一个项目合作，吸引和留住STEM领域的低收入学生。该项目的目标是开发计算模型，从韵律线索和不流畅中提取信息，用于各种口语处理应用。该方法利用语音预期声学动力学预测器中的多尺度上下文，以便自动识别非典型计时或夸张的区域。具体来说，它使用具有并行文本和声音输入的深度神经网络来表示局部动态，并结合点过程模型来表征非典型事件的全局速率。语言分析和群体感知研究用于确定信息承载的异常类型（相对于语言处理中应忽略的噪声），从而改进语音理解模型。实验利用各种数据来源来评估适应策略并确保研究结果的普遍性。计算模型的评估是在多个下游应用的背景下进行的，以便广泛地探索潜在的贡献。

Spoken language encodes significant information in pitch and energy dynamics (prosody) and in disfluencies (self-edits) that human listeners use to understand a talker's meaning and the social/emotional context. Due to a lack of adequate models of these phenomena, current speech processing systems make little use of this information. This project tackles modeling limitations by focusing on unexpected speech phenomena, assuming that these events often carry the most valuable information, and by working with speech from a variety of social contexts. The work has applications that range from literacy assessment to improved human-computer interaction. Further, understanding the communicative role of different disfluencies in non-clinical speech will lead to more accurate clinical diagnoses. Educational aspects aim at broad exposure of the research methods to a diverse group of students at all academic levels through short courses, student TED talks, and work with a UW program for attracting and retaining low income students in STEM fields.The goal of this project is to develop computational models that extract information from prosodic cues and disfluencies for use in a variety of spoken language processing applications. The approach leverages multiscale context in predictors of expected acoustic dynamics of speech in order to automatically identify regions of atypical timing or exaggeration. Specifically, it uses deep neural networks with parallel text and acoustic inputs to represent local dynamics in combination with point process models to characterize global rates of atypical events. Linguistic analyses and crowd-sourced perception studies are used to determine types of anomalies that are information bearing (vs. noise that should be ignored in language processing), leading to improved speech understanding models. Experiments make use of a variety of data sources to assess adaptation strategies and ensure generalizability of findings. Evaluation of computational models is in the context of multiple downstream applications in order to broadly explore potential contributions.