Multimodal communication and cognition: The role of gesture in language processing and word learning in individuals with traumatic brain injury

项目摘要 沟通是多模态的，包括语音和手势。当人们说话时， （手和手臂的自发运动）可以直观地描述言语中传达的信息，但通常 传达言语中没有的独特信息。例如，说话者可能会说，“我搜索了一个 新的食谱，”而作出打字的手势，传达只有在手势，发言人搜索在线，而不是 而不是通过一本烹饪书。听者必须将来自言语和视觉空间信息的语言信息结合起来 从手势到生成信息的集成表示。手势的好处 交流和认知在神经型个体中有充分的记载。例如，手势可以改善 理解和记忆口语信息，促进单词学习，对学术至关重要的能力 职业成功。然而，手势在临床人群中尚未受到同样的关注， 认知交流障碍，如创伤性脑损伤（TBI）。在这份报告中，我们研究了 手势的好处延伸到TBI患者，或者如果他们的缺陷的性质阻止手势的 促进沟通和认知的作用。使用一种新的方法， 言语语言病理学，手势研究，心理语言学和神经心理学，我们测试的能力 TBI患者在多模态语言处理和单词学习过程中使用手势， 实验目的1（实验1和2）研究TBI患者的整合能力 在多模态语言处理期间从语音和手势获得信息。实验1测试了 观察叙述者在随后复述故事时的手势，以确定TBI患者是否 报告信息提供独特的姿态，并将其整合到他们的代表性的故事， 时间实验2使用眼动追踪来确定TBI患者是否可以使用从手势到手势的信息。 解决快速在线语言处理过程中视觉世界范式中的指称歧义。目的2 （实验3）调查TBI个体是否从观察和产生手势中受益。 单词学习探索性目标3考察了言语-手势整合与工作之间的关系 记忆能力，以探索手势处理的个体差异，并为未来的验证性研究提供信息。 研究手势沿着说话对于提供语言的生态有效评估至关重要， 更接近描述和丰富日常生活的真实世界通信环境。 这项拟议中的研究将直接推动临床人群中手势的研究， 深入了解在多模态语言处理的背景下整合语音和手势的能力， 测试手势是否可以用来支持TBI患者的新学习。本研究 代表了TBI研究的新方向，并有望为理论提供新颖和有影响力的贡献 多模式沟通和TBI后认知沟通缺陷的性质。

PROJECT SUMMARY Communication is multimodal, containing speech and gesture. When people talk, co-speech gestures (spontaneous movements of the hands and arms) can visually depict information conveyed in speech but often communicate unique information not conveyed in speech. For example, a speaker might say, “I searched for a new recipe,” while making a typing gesture, conveying only in gesture that the speaker searched online rather than through a cookbook. Listeners must bind linguistic information from speech and visuospatial information from gesture to generate an integrated representation of a message. The benefits of gesture for communication and cognition are well-documented in neurotypical individuals. For example, gesture improves comprehension and memory for spoken information and facilitates word learning, abilities critical for academic and vocational success. However, gesture has not yet received the same attention in clinical populations with cognitive-communication disorders, such as traumatic brain injury (TBI). In this proposal, we examine whether the benefits of gesture extend to individuals with TBI, or if the very nature of their deficits prevent gesture’s facilitatory role in communication and cognition. Using a novel approach combining methods and theory from speech-language pathology, gesture studies, psycholinguistics, and neuropsychology, we test the ability of individuals with TBI to use gesture during multimodal language processing and word learning across three experiments. Aim 1 (Experiments 1 and 2) investigates the ability of individuals with TBI to integrate information from speech and gesture during multimodal language processing. Experiment 1 tests the effect of observing a narrator’s gestures on subsequent retellings of stories to determine whether individuals with TBI report information provided uniquely in gesture and integrate it into their representation of the stories across time. Experiment 2 uses eye-tracking to determine if individuals with TBI can use information from gesture to resolve referential ambiguity in a visual-world paradigm during rapid on-line language processing. Aim 2 (Experiment 3) investigates whether individuals with TBI benefit from observing and producing gesture during word learning. Exploratory Aim 3 examines the relation between speech-gesture integration and working memory abilities to explore individual differences in gesture processing and inform future confirmatory studies. Studying gesture along with speech is critical for providing ecologically valid assessments of language that more closely approximate the real-world communication contexts that characterize and enrich everyday life. The proposed research will directly advance the study of gesture in clinical populations by providing new insight into the ability to integrate speech and gesture in the context of multimodal language processing and testing whether gesture can be leveraged to support new learning in individuals with TBI. This research represents a new direction in TBI research and promises to offer novel and impactful contributions to theories of multimodal communication and to the nature of cognitive-communication deficits after TBI.