Doctoral Dissertation Research: Integrating face and acoustic cues during native- and nonnative-accented speech processing: The role of face cue predictability

在不同语言环境中生活和活动的人可能会改变他们的说话方式，这取决于他们与谁交谈。根据语境的不同，单个说话者可能会发出带有母语口音的讲话，也可能会发出带有非母语口音的讲话。例如，一个在西班牙语家庭长大的西班牙语-英语双语青少年在与父母和兄弟姐妹交谈时可能会使用带有西班牙口音的英语，但在学校与朋友交谈时可能会使用带有美国口音的英语。一项研究考察了听者如何处理带有外国口音的语音，将听者对带有外国口音的人的理解与他们对带有本国口音的人的理解进行了比较。很少有研究考察听话者如何理解可以说外国口音或母语口音的说话者，换句话说，当听者无法预测说话者将使用哪种口音时。随着我们社会的语言环境变得越来越多样化，研究听者对同一人产生的不同类型口音的处理是很重要的。听众使用哪些线索来预测说话人的口音，以及他们如何使用关于说话人身份的信息(面孔、口音)来处理他们的语音？本项目使用行为和神经认知(EEG)方法来研究以下问题：1)听者是否使用脸部线索来预测即将到来的语音口音，以及他们如何在在线母语口音和非母语口音的语音处理过程中整合脸部线索和语音口音？2)在2a)单词和2b)句子水平上，脸部线索预测即将到来的语音口音的可靠性如何影响母语口音和非母语口音语音处理的行为和神经相关性？在两个实验中，参与者学习说话人的口音(S)，然后在记录脑电的同时完成一项面孔线索词汇判断任务(实验2A)或一项面孔线索句子加工任务(实验2B)。当播放演讲时，面部提示同时存在。重要的是，在面孔线索的开始和语音信号的开始之间将存在时间延迟，以检验面孔线索的可预测性效果。在另外两个实验中，参与者完成了与实验2A和2B相同的脑电任务，但没有面部提示，以检查只处理口语单词(实验1A)和处理句子(实验1B)的神经认知机制。通过研究听众如何理解在不同口音之间转换的说话者，该项目将为在日益多样化的语言社会中的日常交流模式提供新的见解。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Individuals who live and operate in linguistically diverse contexts may change the way they speak depending on whom they are speaking to. Depending on the contextual situation, a single speaker may produce native-accented speech or nonnative-accented speech. For example, a Spanish-English bilingual teenager growing up in a Spanish-speaking household may use Spanish-accented English when speaking with their parents and siblings, but may use American-accented English when speaking with friends at school. Research that examines how listeners process foreign-accented speech has compared listeners' comprehension of foreign-accented speakers versus their comprehension of native-accented speakers. Few studies have examined how listeners comprehend speakers who can speak with either a foreign or a native accent, in other words, when listeners cannot predict which accent the speaker will use. As the language landscape becomes more diverse in our society, it is important to examine listeners' processing of different types of accented speech as produced by the same individual. Which cues do listeners use to predict a speaker’s accent and how do they use information about the speaker’s identity (face, accent) to process their speech? This project employs behavioral and neurocognitive (EEG) methods to examine the following questions: 1) Do listeners use face cues to predict the upcoming speech accent and how do they integrate face cues and speech accent during online native-accented and nonnative-accented speech processing? 2) How does the reliability of face cues in predicting the upcoming speech accent affect behavioral and neural correlates of native-accented and nonnative-accented speech processing, at the 2a) word level and 2b) sentence level? In two experiments, participants learn about the speakers’ accent(s), then complete a face-cued go/no-go lexical decision task (Experiment 2A) or a face-cued sentence processing task (Experiment 2B) while EEG is recorded. The face cue is concurrently present as the speech is played. Crucially, there will be a time delay between the onset of the face cue and the onset of the speech signal to examine face cue predictability effects. In two additional experiments, participants complete the same EEG tasks as Experiments 2A and 2B, but without a face cue, to examine the neurocognitive mechanisms related to the processing of spoken words alone (Experiment 1A) and sentences (Experiment 1B). By studying how listeners understand speakers who switch between accents, this project will provide novel insights in patterns of everyday communication in an increasingly linguistically diverse society.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.