Making sense from the hands and mouth: multimodal integration in spoken and signed languages

点击翻译按钮获取中文摘要

Successful face-to-face communication involves integrating multiple types of information. In sign languages, linguistic information is provided by the hands, but also by the body, face and mouth. In spoken languages, the mouth provides linguistic information, an acoustic signal coupled with visual information reflecting not only mouth movements but also gestures. In both signed and spoken languages, the information provided by different articulators is tightly synchronised, facilitating comprehension. Our understanding of signed and spoken language is typically limited by a focus on a single, dominant modality (hands in British Sign Language (BSL); speech in English), despite the pervasiveness of simultaneous expression in multiple modalities and despite evidence that the "subordinate" modalities (mouth in sign languages; gestures in spoken language) play important roles in communication. This project investigates how information from mouth and hands for signers and speakers is integrated, testing how integration affects language comprehension in signed and spoken languages, and the extent to which common neural systems subserve integration. To address these issues, the applicant will develop behavioural and functional MRI studies in parallel for BSL and spoken English to assess the consequences of content and timing mismatches between hands and mouth in comprehension. Behavioural experiments take the novel approach of using digitally manipulated BSL and English video stimuli with head, torso and hands all fully visible, to test the consequences of mismatch between content expressed by the hands and mouth, as well as temporal asynchrony. Functional MRI studies will assess the neural substrate of system(s) engaged in integrating information from hands and mouth in BSL and English. By conducting highly parallel experiments in BSL and English, the behavioural and neural consequences of integrating two visual linguistic modalities (hand movements and mouthing in BSL) can be directly contrasted to the consequences of integrating linguistic information from different modalities (acoustic speech signal and mouth movements perceived visually in English) with paralinguistic information (co-speech gesture). The project further addresses whether the same brain networks may be involved in hand-mouth integration in BSL, and speech-gesture in English, despite differences in the roles of these articulators, and whether BSL-English differences can be explained in terms of deafness or sign language experience.The applicant has strong behavioural experimentation skills in spoken and signed languages as reflected by his more than 25 publications in peer reviewed journals (7 as first author); and has preliminary skills in neuroscience as well as knowledge of BSL. A significant training component will be extending theoretical and methodological neuroscientific skills under the guidance of mentor Prof Marty Sereno. As well as being an internationally recognised researcher in cognitive neuroscience, Professor Sereno has specific expertise in the neural systems underlying different modalities of perception and their interaction. University College London is uniquely endowed with the resources and personnel to conduct the proposed studies to a high standard and at a low cost. UCL is committed to supporting multimodal communication as a research priority; the Deafness, Cognition and Language Research Centre offers filming and editing suites, extensive contacts in the Deaf community for participant recruitment and knowledge transfer as well as a Deaf community liaison officer; and the Birkbeck-UCL Centre for NeuroImaging has facilities for neuroimaging and existing protocols for scanning deaf participants.The proposed studies not only have broad potential academic impact in areas including psychology, linguistics and neuroscience, but also have potential impact in educational practice, language rehabilitation and communication technology.