Collaborative research: An integrated model of phonetic analysis and lexical analysis based on individual acoustic cues to features

认知科学和神经科学中最大的谜团之一是，人类如何在对任何给定的语音或单词产生的精确声学产生极端差异的情况下，实现强大的语音感知。例如，人们可以为相同的元音产生不同的声学效果，而在其他情况下，两个不同元音的声学效果可能几乎相同。声学模式也会随着声音的发声速度而变化。听者还可能感觉到由于语音发音的大幅减少而实际上并未产生的声音(例如，“Do‘t You”中的“t”和“y”发音通常被简化为“doncha”)。大多数理论认为，听者通过严格按顺序提取辅音和元音来识别连续言语中的单词。然而，之前的研究未能找到证据证明声音信号中的不变线索可以让听者提取重要信息。这个项目使用了一种研究语言处理的新工具，Lexi(用于语言事件提取和解释)，来检验这样一个假设，即辅音和元音的单个声学提示实际上可以从信号中提取出来，并可以用来确定说话人的意图。当语音的某些声学线索被修改或丢失时，Lexi可以检测剩余的线索，并将它们作为预期声音和单词的证据进行评估。这项研究具有潜在的广泛的社会效益，包括优化人机交互，以适应言语障碍或口音重音中出现的非典型言语模式。该项目通过实验和计算研究的实践经验，支持1-2名博士生和8-10名本科生的培训。所有数据，包括计算模型的代码、Lexi系统和标记为声学线索的语音数据库将通过开放科学框架公开获得；所有出版物的预印本将在心理Arxiv和NSF-PAR公开获得。这个跨学科项目结合了信号分析、心理语言学实验和计算建模，以(1)调查声学线索在不同环境中变化的方式，(2)实验测试听者如何通过语音的分布式学习使用这些线索，以及(3)使用计算建模来评估关于听者如何识别口语单词的相互竞争的理论。这项工作将识别信号中的线索模式，听者用来识别发音的大幅减少，并将在实验中测试听者如何跟踪这种系统的变化。这些知识将被用来模拟听众如何“调谐”到说话者产生语音的不同方式。通过使用Lexi检测到的线索作为相互竞争的单词识别模型的输入，这项工作提供了一个机会，来研究人类对大量口语单词的语音识别的细粒度时间进程；这是一项重要的创新，因为大多数语音的认知模型不直接与语音输入一起工作。理论上的好处包括对基于提示的单词识别模型进行了强有力的测试，并开发了工具以允许几乎任何语音识别模型在真实语音输入上工作，这对优化自动语音识别具有实际意义。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

One of the greatest mysteries in the cognitive and neural sciences is how humans achieve robust speech perception given extreme variation in the precise acoustics produced for any given speech sound or word. For example, people can produce different acoustics for the same vowel sound, while in other cases the acoustics for two different vowels may be nearly identical. The acoustic patterns also change depending on the rate at which the sounds are spoken. Listeners may also perceive a sound that was not actually produced due to massive reductions in speech pronunciation (e.g., the "t" and "y" sounds in "don't you" are often reduced to "doncha"). Most theories assume that listeners recognize words in continuous speech by extracting consonants and vowels in a strictly sequential order. However, previous research has failed to find evidence for invariant cues in the acoustic signal that would allow listeners to extract the important information. This project uses a new tool for the study of language processing, LEXI (for Linguistic-Event EXtraction and Interpretation), to test the hypothesis that individual acoustic cues for consonants and vowels can in fact be extracted from the signal and can be used to determine the speaker's intended words. When some acoustic cues for speech sounds are modified or missing, LEXI can detect the remaining cues and evaluate them as evidence for the intended sounds and words. This research has potentially broad societal benefits, including optimization of human-machine interactions to accommodate atypical speech patterns seen in speech disorders or accented speech. This project supports training of 1-2 doctoral students and 8-10 undergraduate students through hands-on experience in experimental and computational research. All data, including code for computational models, the LEXI system, and speech databases labeled for acoustic cues will be publicly available through the Open Science Framework; preprints of all publications will be publicly available at PsyArxiv and NSF-PAR.This interdisciplinary project unites signal analysis, psycholinguistic experimentation, and computational modeling to (1) survey the ways that acoustic cues vary in different contexts, (2) experimentally test how listeners use these cues through distributional learning for speech, and (3) use computational modeling to evaluate competing theories of how listeners recognize spoken words. The work will identify cue patterns in the signal that listeners use to recognize massive reductions in pronunciation and will experimentally test how listeners keep track of this systematic variation. This knowledge will be used to model how listeners "tune in" to the different ways speakers produce speech sounds. By using cues detected by LEXI as input to competing models of word recognition, the work provides an opportunity to examine the fine-grained time course of human speech recognition with large sets of spoken words; this is an important innovation because most cognitive models of speech do not work with speech input directly. Theoretical benefits include a strong test of the cue-based model of word recognition and the development of tools to allow virtually any model of speech recognition to work on real speech input, with practical implications for optimizing automatic speech recognition.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.