Doctoral Dissertation Research: Integrating face and acoustic cues during native- and nonnative-accented speech processing: The role of face cue predictability

生活和工作在不同语言环境中的人可能会根据他们说话的对象而改变他们说话的方式。根据上下文的情况，单个说话者可以产生本地口音的语音或非本地口音的语音。例如，在西班牙语家庭长大的西班牙语-英语双语青少年在与父母和兄弟姐妹交谈时可能使用西班牙口音的英语，但在学校与朋友交谈时可能使用美国口音的英语。一项研究探讨了听众如何处理外国口音的语音，比较了听众对外国口音说话者的理解与他们对本土口音说话者的理解。很少有研究探讨听众如何理解说话者可以用外国或本地口音说话，换句话说，当听众无法预测说话者将使用哪种口音时。随着社会语言的多样化，研究听者对同一个人发出的不同类型的口音语音的处理就显得尤为重要。听众使用哪些线索来预测说话者的口音，他们如何使用有关说话者身份（面部，口音）的信息来处理他们的讲话？本研究采用行为和神经认知（EEG）的方法来研究以下问题：1）在在线的本地口音和非本地口音语音处理过程中，听者是否使用面部线索来预测即将到来的语音口音，以及他们如何整合面部线索和语音口音？2)在2a）单词水平和2b）句子水平上，面部线索在预测即将到来的语音口音中的可靠性如何影响本地口音和非本地口音语音处理的行为和神经相关性？在两个实验中，被试学习说话人的口音，然后完成面部线索的go/no-go词汇判断任务（实验2A）或面部线索的句子加工任务（实验2B），同时记录EEG。当播放语音时，面部提示同时存在。至关重要的是，在面部线索的开始和语音信号的开始之间将存在时间延迟，以检查面部线索的可预测性效果。在另外两个实验中，参与者完成与实验2A和2B相同的EEG任务，但没有面部提示，以检查与单独的口语单词（实验1A）和句子（实验1B）处理相关的神经认知机制。通过研究听众如何理解在不同口音之间转换的说话者，该项目将为语言日益多样化的社会中的日常交流模式提供新的见解。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Individuals who live and operate in linguistically diverse contexts may change the way they speak depending on whom they are speaking to. Depending on the contextual situation, a single speaker may produce native-accented speech or nonnative-accented speech. For example, a Spanish-English bilingual teenager growing up in a Spanish-speaking household may use Spanish-accented English when speaking with their parents and siblings, but may use American-accented English when speaking with friends at school. Research that examines how listeners process foreign-accented speech has compared listeners' comprehension of foreign-accented speakers versus their comprehension of native-accented speakers. Few studies have examined how listeners comprehend speakers who can speak with either a foreign or a native accent, in other words, when listeners cannot predict which accent the speaker will use. As the language landscape becomes more diverse in our society, it is important to examine listeners' processing of different types of accented speech as produced by the same individual. Which cues do listeners use to predict a speaker’s accent and how do they use information about the speaker’s identity (face, accent) to process their speech? This project employs behavioral and neurocognitive (EEG) methods to examine the following questions: 1) Do listeners use face cues to predict the upcoming speech accent and how do they integrate face cues and speech accent during online native-accented and nonnative-accented speech processing? 2) How does the reliability of face cues in predicting the upcoming speech accent affect behavioral and neural correlates of native-accented and nonnative-accented speech processing, at the 2a) word level and 2b) sentence level? In two experiments, participants learn about the speakers’ accent(s), then complete a face-cued go/no-go lexical decision task (Experiment 2A) or a face-cued sentence processing task (Experiment 2B) while EEG is recorded. The face cue is concurrently present as the speech is played. Crucially, there will be a time delay between the onset of the face cue and the onset of the speech signal to examine face cue predictability effects. In two additional experiments, participants complete the same EEG tasks as Experiments 2A and 2B, but without a face cue, to examine the neurocognitive mechanisms related to the processing of spoken words alone (Experiment 1A) and sentences (Experiment 1B). By studying how listeners understand speakers who switch between accents, this project will provide novel insights in patterns of everyday communication in an increasingly linguistically diverse society.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.