Doctoral Dissertation Research: Integrating face and acoustic cues during native- and nonnative-accented speech processing: The role of face cue predictability

生活和工作在不同语言环境中的人可能会根据说话对象的不同而改变说话方式。根据上下文情况，单个说话者可能会说出母语口音或非母语口音的话。例如，在说西班牙语的家庭中长大的西班牙语-英语双语青少年在与父母和兄弟姐妹交谈时可能会使用西班牙口音的英语，但在学校与朋友交谈时可能会使用美国口音的英语。一项关于听者如何处理外国口音的研究比较了听者对外国口音说话者的理解和对本国口音说话者的理解。很少有研究调查听者是如何理解既带外国口音又带本国口音的说话者的，换句话说，当听者无法预测说话者会使用哪种口音时。随着我们社会的语言环境变得越来越多样化，研究听者对同一个人发出的不同类型的口音语音的处理是很重要的。听众使用哪些线索来预测说话人的口音？他们如何使用说话人的身份信息（面孔、口音）来处理他们的讲话？该项目采用行为和神经认知（EEG）方法来研究以下问题：1)听者是否使用面部线索来预测即将到来的语音口音，以及他们如何在在线的母语口音和非母语口音语音处理中整合面部线索和语音口音？2)在2a)单词水平和2b)句子水平上，面部线索预测即将到来的语音口音的可靠性如何影响母语和非母语口音语音处理的行为和神经相关？在两个实验中，参与者先了解说话者的口音，然后在记录脑电图的同时完成一项面部提示的“走/不走”词汇决策任务（实验2A）或一项面部提示的句子处理任务（实验2B）。当演讲播放时，面部提示同时出现。至关重要的是，在人脸提示的出现和语音信号的出现之间会有一个时间延迟，以检验人脸提示的可预测性效果。在另外两个实验中，参与者完成了与实验2A和2B相同的脑电图任务，但没有面部提示，以检查与单独口语（实验1A）和句子（实验1B）处理相关的神经认知机制。通过研究听者如何理解说话人在不同口音之间的转换，该项目将为在语言日益多样化的社会中日常交流模式提供新的见解。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Individuals who live and operate in linguistically diverse contexts may change the way they speak depending on whom they are speaking to. Depending on the contextual situation, a single speaker may produce native-accented speech or nonnative-accented speech. For example, a Spanish-English bilingual teenager growing up in a Spanish-speaking household may use Spanish-accented English when speaking with their parents and siblings, but may use American-accented English when speaking with friends at school. Research that examines how listeners process foreign-accented speech has compared listeners' comprehension of foreign-accented speakers versus their comprehension of native-accented speakers. Few studies have examined how listeners comprehend speakers who can speak with either a foreign or a native accent, in other words, when listeners cannot predict which accent the speaker will use. As the language landscape becomes more diverse in our society, it is important to examine listeners' processing of different types of accented speech as produced by the same individual. Which cues do listeners use to predict a speaker’s accent and how do they use information about the speaker’s identity (face, accent) to process their speech? This project employs behavioral and neurocognitive (EEG) methods to examine the following questions: 1) Do listeners use face cues to predict the upcoming speech accent and how do they integrate face cues and speech accent during online native-accented and nonnative-accented speech processing? 2) How does the reliability of face cues in predicting the upcoming speech accent affect behavioral and neural correlates of native-accented and nonnative-accented speech processing, at the 2a) word level and 2b) sentence level? In two experiments, participants learn about the speakers’ accent(s), then complete a face-cued go/no-go lexical decision task (Experiment 2A) or a face-cued sentence processing task (Experiment 2B) while EEG is recorded. The face cue is concurrently present as the speech is played. Crucially, there will be a time delay between the onset of the face cue and the onset of the speech signal to examine face cue predictability effects. In two additional experiments, participants complete the same EEG tasks as Experiments 2A and 2B, but without a face cue, to examine the neurocognitive mechanisms related to the processing of spoken words alone (Experiment 1A) and sentences (Experiment 1B). By studying how listeners understand speakers who switch between accents, this project will provide novel insights in patterns of everyday communication in an increasingly linguistically diverse society.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.