Collaborative research: An integrated model of phonetic analysis and lexical access based on individual acoustic cues to features

认知和神经科学中最大的谜团之一是，在任何给定的语音或单词产生的精确声学变化极大的情况下，人类如何实现强大的语音感知。例如，人们可以对相同的元音产生不同的声学效果，而在其他情况下，两个不同元音的声学效果可能几乎相同。声学模式也会根据声音的发音速度而变化。 听众还可能感知到由于语音发音的大量减少而实际上没有产生的声音（例如，“don 't you”中的“t”和“y”音通常被简化为“doncha”）。大多数理论认为，听者通过严格按顺序提取辅音和元音来识别连续语音中的单词。然而，以前的研究未能找到证据，在声学信号中的不变线索，这将使听众提取的重要信息。该项目使用了一种新的语言处理研究工具，LEXI（用于语言事件提取和解释），以测试假设，即辅音和元音的单个声学线索实际上可以从信号中提取，并可用于确定说话者的意图。当语音的某些声学线索被修改或丢失时，LEXI可以检测剩余的线索，并将其作为预期声音和单词的证据进行评估。这项研究具有潜在的广泛的社会效益，包括优化人机交互，以适应言语障碍或口音言语中的非典型言语模式。该项目通过实验和计算研究的实践经验，支持1-2名博士生和8-10名本科生的培训。所有数据，包括计算模型的代码、LEXI系统和标记为声学线索的语音数据库，都将通过开放科学框架公开提供;所有出版物的预印本将在PsyArxiv和NSF-PAR上公开提供。这个跨学科的项目将信号分析，心理语言学实验和计算建模结合起来，以（1）调查声学线索在不同背景下的变化方式，（2）通过实验测试听众如何通过语音的分布式学习来使用这些线索，（3）使用计算建模来评估听众如何识别口语单词的竞争理论。这项工作将确定信号中的提示模式，听众使用这些模式来识别发音的大量减少，并将通过实验测试听众如何跟踪这种系统性变化。这些知识将被用来模拟听众如何“调谐”到扬声器产生语音的不同方式。通过使用LEXI检测到的线索作为单词识别的竞争模型的输入，这项工作提供了一个机会来研究人类语音识别的细粒度时间过程与大量的口语单词;这是一个重要的创新，因为大多数语音认知模型不直接与语音输入。理论上的好处包括对基于提示的单词识别模型的强有力的测试，以及开发工具，使几乎任何语音识别模型都能在真实的语音输入上工作，并对优化自动语音识别产生实际影响。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

One of the greatest mysteries in the cognitive and neural sciences is how humans achieve robust speech perception given extreme variation in the precise acoustics produced for any given speech sound or word. For example, people can produce different acoustics for the same vowel sound, while in other cases the acoustics for two different vowels may be nearly identical. The acoustic patterns also change depending on the rate at which the sounds are spoken. Listeners may also perceive a sound that was not actually produced due to massive reductions in speech pronunciation (e.g., the "t" and "y" sounds in "don't you" are often reduced to "doncha"). Most theories assume that listeners recognize words in continuous speech by extracting consonants and vowels in a strictly sequential order. However, previous research has failed to find evidence for invariant cues in the acoustic signal that would allow listeners to extract the important information. This project uses a new tool for the study of language processing, LEXI (for Linguistic-Event EXtraction and Interpretation), to test the hypothesis that individual acoustic cues for consonants and vowels can in fact be extracted from the signal and can be used to determine the speaker's intended words. When some acoustic cues for speech sounds are modified or missing, LEXI can detect the remaining cues and evaluate them as evidence for the intended sounds and words. This research has potentially broad societal benefits, including optimization of human-machine interactions to accommodate atypical speech patterns seen in speech disorders or accented speech. This project supports training of 1-2 doctoral students and 8-10 undergraduate students through hands-on experience in experimental and computational research. All data, including code for computational models, the LEXI system, and speech databases labeled for acoustic cues will be publicly available through the Open Science Framework; preprints of all publications will be publicly available at PsyArxiv and NSF-PAR.This interdisciplinary project unites signal analysis, psycholinguistic experimentation, and computational modeling to (1) survey the ways that acoustic cues vary in different contexts, (2) experimentally test how listeners use these cues through distributional learning for speech, and (3) use computational modeling to evaluate competing theories of how listeners recognize spoken words. The work will identify cue patterns in the signal that listeners use to recognize massive reductions in pronunciation and will experimentally test how listeners keep track of this systematic variation. This knowledge will be used to model how listeners "tune in" to the different ways speakers produce speech sounds. By using cues detected by LEXI as input to competing models of word recognition, the work provides an opportunity to examine the fine-grained time course of human speech recognition with large sets of spoken words; this is an important innovation because most cognitive models of speech do not work with speech input directly. Theoretical benefits include a strong test of the cue-based model of word recognition and the development of tools to allow virtually any model of speech recognition to work on real speech input, with practical implications for optimizing automatic speech recognition.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.