Making sense from the hands and mouth: multimodal integration in spoken and signed languages

成功的面对面交流包括整合多种类型的信息。在手语中，语言信息是由手提供的，但也由身体、面部和嘴巴提供。在口语中，嘴提供语言信息，这是一种与视觉信息相结合的声音信号，不仅反映了嘴的运动，还反映了手势。在手语和口语中，不同发音者提供的信息是紧密同步的，有助于理解。我们对手语和口语的理解通常局限于单一的、占主导地位的情态(英国手语中的手；尽管同时以多种方式表达的现象普遍存在，尽管有证据表明“从属”方式（手语中的嘴部；口语中的手势）在交流中发挥着重要作用。该项目研究了来自手语者和说话者的嘴和手的信息是如何整合的，测试整合如何影响手语和口语的语言理解，以及共同神经系统在多大程度上支持整合。为了解决这些问题，申请人将对车贴语和口语进行并行的行为和功能MRI研究，以评估手和嘴在理解内容和时间上的不匹配的后果。行为实验采用一种新颖的方法，使用数字操纵的车贴语和英语视频刺激，头部、躯干和手都完全可见，以测试手和嘴表达的内容不匹配以及时间异步的后果。功能性核磁共振研究将评估系统的神经基质，这些系统参与整合来自手和嘴的信息。通过在车贴语和英语中进行高度平行的实验，可以直接对比车贴语中两种视觉语言模态（手部运动和嘴部运动）整合的行为和神经后果，以及将不同模态的语言信息（英语中视觉感知的语音信号和嘴部运动）与副语言信息（同语手势）整合的后果。该项目进一步研究了是否同样的大脑网络可能参与车贴语中的手-口整合，以及英语中的言语-手势，尽管这些发音器的作用不同，以及车贴语和英语的差异是否可以用耳聋或手语经验来解释。申请人在同行评审的期刊上发表超过25篇论文（其中7篇为第一作者），表明申请人在口语和手语方面有很强的行为实验技巧；具备初步的神经科学技能和车贴语知识。在导师Marty Sereno教授的指导下，一个重要的培训内容将是扩展神经科学的理论和方法技能。作为一名国际公认的认知神经科学研究人员，Sereno教授在不同感知模式及其相互作用的神经系统方面有着专门的专业知识。伦敦大学学院拥有独特的资源和人员，以高标准和低成本开展拟议的研究。伦敦大学学院致力于支持多模式通信作为研究重点；耳聋、认知和语言研究中心提供拍摄和剪辑套房，在聋人社区广泛联系，以招募参与者和知识转移，以及聋人社区联络官；伯克贝克-伦敦大学学院神经成像中心拥有神经成像设备和现有的扫描聋人参与者的协议。这些研究不仅在心理学、语言学和神经科学等领域具有广泛的潜在学术影响，而且在教育实践、语言康复和通信技术等领域也具有潜在的影响。

Successful face-to-face communication involves integrating multiple types of information. In sign languages, linguistic information is provided by the hands, but also by the body, face and mouth. In spoken languages, the mouth provides linguistic information, an acoustic signal coupled with visual information reflecting not only mouth movements but also gestures. In both signed and spoken languages, the information provided by different articulators is tightly synchronised, facilitating comprehension. Our understanding of signed and spoken language is typically limited by a focus on a single, dominant modality (hands in British Sign Language (BSL); speech in English), despite the pervasiveness of simultaneous expression in multiple modalities and despite evidence that the "subordinate" modalities (mouth in sign languages; gestures in spoken language) play important roles in communication. This project investigates how information from mouth and hands for signers and speakers is integrated, testing how integration affects language comprehension in signed and spoken languages, and the extent to which common neural systems subserve integration. To address these issues, the applicant will develop behavioural and functional MRI studies in parallel for BSL and spoken English to assess the consequences of content and timing mismatches between hands and mouth in comprehension. Behavioural experiments take the novel approach of using digitally manipulated BSL and English video stimuli with head, torso and hands all fully visible, to test the consequences of mismatch between content expressed by the hands and mouth, as well as temporal asynchrony. Functional MRI studies will assess the neural substrate of system(s) engaged in integrating information from hands and mouth in BSL and English. By conducting highly parallel experiments in BSL and English, the behavioural and neural consequences of integrating two visual linguistic modalities (hand movements and mouthing in BSL) can be directly contrasted to the consequences of integrating linguistic information from different modalities (acoustic speech signal and mouth movements perceived visually in English) with paralinguistic information (co-speech gesture). The project further addresses whether the same brain networks may be involved in hand-mouth integration in BSL, and speech-gesture in English, despite differences in the roles of these articulators, and whether BSL-English differences can be explained in terms of deafness or sign language experience.The applicant has strong behavioural experimentation skills in spoken and signed languages as reflected by his more than 25 publications in peer reviewed journals (7 as first author); and has preliminary skills in neuroscience as well as knowledge of BSL. A significant training component will be extending theoretical and methodological neuroscientific skills under the guidance of mentor Prof Marty Sereno. As well as being an internationally recognised researcher in cognitive neuroscience, Professor Sereno has specific expertise in the neural systems underlying different modalities of perception and their interaction. University College London is uniquely endowed with the resources and personnel to conduct the proposed studies to a high standard and at a low cost. UCL is committed to supporting multimodal communication as a research priority; the Deafness, Cognition and Language Research Centre offers filming and editing suites, extensive contacts in the Deaf community for participant recruitment and knowledge transfer as well as a Deaf community liaison officer; and the Birkbeck-UCL Centre for NeuroImaging has facilities for neuroimaging and existing protocols for scanning deaf participants.The proposed studies not only have broad potential academic impact in areas including psychology, linguistics and neuroscience, but also have potential impact in educational practice, language rehabilitation and communication technology.