RI: Small: Modeling Idiosyncrasies of Speech for Automatic Spoken Language Processing

口语以音调和能量动态(韵律)和不流畅(自我编辑)的形式编码重要信息，人类听者利用这些信息来理解说话者的意思和社会/情感背景。由于缺乏对这些现象的适当模型，目前的语音处理系统很少使用这些信息。这个项目通过关注意想不到的言语现象，假设这些事件往往携带最有价值的信息，并通过处理来自各种社会背景的言语，来解决建模限制。这项工作的应用范围从识字评估到改进的人机交互。此外，了解非临床言语中不同不流利性的交际作用将有助于更准确的临床诊断。教育方面的目标是通过短期课程，学生TED演讲，以及与UW项目合作，在STEM领域吸引和留住低收入学生，向不同学术水平的学生广泛展示研究方法。该项目的目标是开发计算模型，从韵律线索和不流利的信息中提取信息，用于各种口语处理应用程序。该方法利用语音的预期声学动态的预测器中的多尺度上下文，以便自动识别非典型定时或夸张的区域。具体地说，它使用具有并行文本和声音输入的深度神经网络来表示局部动力学，并结合点过程模型来表征非典型事件的全局比率。语言分析和众包感知研究被用来确定承载信息的异常类型(与语言处理中应该忽略的噪声)，从而导致改进的语音理解模型。实验利用各种数据来源来评估适应策略，并确保研究结果的概括性。对计算模型的评估是在多个下游应用的背景下进行的，以便广泛地探索潜在的贡献。

Spoken language encodes significant information in pitch and energy dynamics (prosody) and in disfluencies (self-edits) that human listeners use to understand a talker's meaning and the social/emotional context. Due to a lack of adequate models of these phenomena, current speech processing systems make little use of this information. This project tackles modeling limitations by focusing on unexpected speech phenomena, assuming that these events often carry the most valuable information, and by working with speech from a variety of social contexts. The work has applications that range from literacy assessment to improved human-computer interaction. Further, understanding the communicative role of different disfluencies in non-clinical speech will lead to more accurate clinical diagnoses. Educational aspects aim at broad exposure of the research methods to a diverse group of students at all academic levels through short courses, student TED talks, and work with a UW program for attracting and retaining low income students in STEM fields.The goal of this project is to develop computational models that extract information from prosodic cues and disfluencies for use in a variety of spoken language processing applications. The approach leverages multiscale context in predictors of expected acoustic dynamics of speech in order to automatically identify regions of atypical timing or exaggeration. Specifically, it uses deep neural networks with parallel text and acoustic inputs to represent local dynamics in combination with point process models to characterize global rates of atypical events. Linguistic analyses and crowd-sourced perception studies are used to determine types of anomalies that are information bearing (vs. noise that should be ignored in language processing), leading to improved speech understanding models. Experiments make use of a variety of data sources to assess adaptation strategies and ensure generalizability of findings. Evaluation of computational models is in the context of multiple downstream applications in order to broadly explore potential contributions.