Making sense from the hands and mouth: multimodal integration in spoken and signed languages

成功的面对面交流需要整合多种类型的信息。在手语中，语言信息是由手提供的，也是通过身体、脸和嘴提供的。在口语中，嘴提供语言信息，这是一种声学信号，与视觉信息相结合，不仅反映嘴巴的动作，还反映手势。在手语和口语中，不同发音者提供的信息是紧密同步的，有助于理解。我们对手语和口语的理解通常局限于一种单一的、占主导地位的情态(英国手语中的手；英语中的言语)，尽管同时表达在多种情态中的现象普遍存在，尽管有证据表明“从属”情态(手语中的嘴；口语中的手势)在交流中发挥着重要作用。这个项目调查了手语者和说话者的嘴巴和手中的信息是如何整合的，测试整合如何影响手语和口语中的语言理解，以及常见神经系统在多大程度上辅助整合。为了解决这些问题，申请者将同时对BSL和英语口语进行行为和功能磁共振研究，以评估理解过程中手和嘴的内容和时间不匹配的后果。行为实验采用了一种新颖的方法，使用数字处理的BSL和英语视频刺激，在头部、躯干和手都完全可见的情况下，测试手和嘴表达的内容之间的不匹配以及时间不同步的后果。功能磁共振成像研究将评估S(音译)系统的神经基础，该系统从事手部和嘴部信息的整合，使用BSL和英语。通过在二语和英语中进行高度平行的实验，整合两种视觉语言形式(二语中的手势和口音)的行为和神经结果可以直接与整合来自不同形式的语言信息(英语中视觉感知的有声语音信号和嘴巴运动)和副语言信息(联合言语手势)的结果形成对比。该项目进一步探讨了相同的大脑网络是否可能参与BSL的手口整合以及英语中的言语-手势，尽管这些发音器的角色不同，以及BSL-英语的差异是否可以用耳聋或手语经验来解释。申请人在同行评议期刊上发表的25多篇论文(第一作者7篇)反映了他在口语和手语方面强大的行为实验技能；并具有神经科学方面的初步技能和BSL的知识。一个重要的培训部分将是在导师Marty Sereno教授的指导下扩展理论和方法神经科学技能。除了是国际公认的认知神经科学研究人员，塞雷诺教授还在不同感知方式及其相互作用的神经系统方面拥有特定的专业知识。伦敦大学学院拥有独特的资源和人员，可以高标准、低成本地开展拟议中的研究。伦敦大学学院致力于支持作为研究重点的多模式交流；聋人、认知和语言研究中心提供拍摄和剪辑套件，在聋人社区中建立广泛的联系，以招募参与者和传递知识，并设立聋人社区联络官；Birkbeck-UCL神经成像中心拥有神经成像设施和现有的扫描聋人参与者的协议。拟议的研究不仅在心理学、语言学和神经科学等领域具有广泛的潜在学术影响，而且在教育实践、语言康复和通信技术方面也具有潜在的影响。

Successful face-to-face communication involves integrating multiple types of information. In sign languages, linguistic information is provided by the hands, but also by the body, face and mouth. In spoken languages, the mouth provides linguistic information, an acoustic signal coupled with visual information reflecting not only mouth movements but also gestures. In both signed and spoken languages, the information provided by different articulators is tightly synchronised, facilitating comprehension. Our understanding of signed and spoken language is typically limited by a focus on a single, dominant modality (hands in British Sign Language (BSL); speech in English), despite the pervasiveness of simultaneous expression in multiple modalities and despite evidence that the "subordinate" modalities (mouth in sign languages; gestures in spoken language) play important roles in communication. This project investigates how information from mouth and hands for signers and speakers is integrated, testing how integration affects language comprehension in signed and spoken languages, and the extent to which common neural systems subserve integration. To address these issues, the applicant will develop behavioural and functional MRI studies in parallel for BSL and spoken English to assess the consequences of content and timing mismatches between hands and mouth in comprehension. Behavioural experiments take the novel approach of using digitally manipulated BSL and English video stimuli with head, torso and hands all fully visible, to test the consequences of mismatch between content expressed by the hands and mouth, as well as temporal asynchrony. Functional MRI studies will assess the neural substrate of system(s) engaged in integrating information from hands and mouth in BSL and English. By conducting highly parallel experiments in BSL and English, the behavioural and neural consequences of integrating two visual linguistic modalities (hand movements and mouthing in BSL) can be directly contrasted to the consequences of integrating linguistic information from different modalities (acoustic speech signal and mouth movements perceived visually in English) with paralinguistic information (co-speech gesture). The project further addresses whether the same brain networks may be involved in hand-mouth integration in BSL, and speech-gesture in English, despite differences in the roles of these articulators, and whether BSL-English differences can be explained in terms of deafness or sign language experience.The applicant has strong behavioural experimentation skills in spoken and signed languages as reflected by his more than 25 publications in peer reviewed journals (7 as first author); and has preliminary skills in neuroscience as well as knowledge of BSL. A significant training component will be extending theoretical and methodological neuroscientific skills under the guidance of mentor Prof Marty Sereno. As well as being an internationally recognised researcher in cognitive neuroscience, Professor Sereno has specific expertise in the neural systems underlying different modalities of perception and their interaction. University College London is uniquely endowed with the resources and personnel to conduct the proposed studies to a high standard and at a low cost. UCL is committed to supporting multimodal communication as a research priority; the Deafness, Cognition and Language Research Centre offers filming and editing suites, extensive contacts in the Deaf community for participant recruitment and knowledge transfer as well as a Deaf community liaison officer; and the Birkbeck-UCL Centre for NeuroImaging has facilities for neuroimaging and existing protocols for scanning deaf participants.The proposed studies not only have broad potential academic impact in areas including psychology, linguistics and neuroscience, but also have potential impact in educational practice, language rehabilitation and communication technology.