Multimodal communication and cognition: The role of gesture in language processing and word learning in individuals with traumatic brain injury

多模式沟通和认知：手势在脑外伤患者语言处理和单词学习中的作用

• 批准号: 10640100
• 负责人: Sharice Clough
• 金额: $ 2.19万
• 依托单位: VANDERBILT UNIVERSITY
• 依托单位国家: 美国
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-01-01 至 2023-08-10
• 项目状态: 已结题
• 来源: https://reporter.nih.gov/project-details/10640100
• 关键词: Adult; Affect; Attention; Binding; Brain; Child; Clinical; Cognition; Cognitive; Cognitive deficits; Communication; Communication impairment; Comprehension; Cookbook; Cues; Detection; Diffuse Brain Injury; Emotional; Face; Future; Gestures; Goals; Hand; Impairment; Individual; Individual Differences; Kinesthesis; Knowledge; Language; Language Disorders; Language Tests; Learning; Life; Linguistics; Link; Memory; Methods; Mind; Motor; Movement; Nature; Neuropsychology; Pathway interactions; Pattern; Performance; Persons; Population; Process; Psycholinguistics; Recipe; Reporting; Research; Resources; Role; Short-Term Memory; Source; Speech; Speech-Language Pathology; System; Testing; Thinking; Time; Training; Traumatic Brain Injury; Visual; Visuospatial; Vocation; arm; behavior measurement; clinical translation; cognitive benefits; cognitive load; episodic memory impairment; experimental study; gaze; improved; innovation; insight; interdisciplinary approach; language processing; lexical retrieval; multimodality; non-verbal; novel; novel strategies; peer; preservation; prevent; procedural memory; recruit; social engagement; success; theories; verbal; visual tracking; white matter; word learning

PROJECT SUMMARY Communication is multimodal, containing speech and gesture. When people talk, co-speech gestures (spontaneous movements of the hands and arms) can visually depict information conveyed in speech but often communicate unique information not conveyed in speech. For example, a speaker might say, “I searched for a new recipe,” while making a typing gesture, conveying only in gesture that the speaker searched online rather than through a cookbook. Listeners must bind linguistic information from speech and visuospatial information from gesture to generate an integrated representation of a message. The benefits of gesture for communication and cognition are well-documented in neurotypical individuals. For example, gesture improves comprehension and memory for spoken information and facilitates word learning, abilities critical for academic and vocational success. However, gesture has not yet received the same attention in clinical populations with cognitive-communication disorders, such as traumatic brain injury (TBI). In this proposal, we examine whether the benefits of gesture extend to individuals with TBI, or if the very nature of their deficits prevent gesture’s facilitatory role in communication and cognition. Using a novel approach combining methods and theory from speech-language pathology, gesture studies, psycholinguistics, and neuropsychology, we test the ability of individuals with TBI to use gesture during multimodal language processing and word learning across three experiments. Aim 1 (Experiments 1 and 2) investigates the ability of individuals with TBI to integrate information from speech and gesture during multimodal language processing. Experiment 1 tests the effect of observing a narrator’s gestures on subsequent retellings of stories to determine whether individuals with TBI report information provided uniquely in gesture and integrate it into their representation of the stories across time. Experiment 2 uses eye-tracking to determine if individuals with TBI can use information from gesture to resolve referential ambiguity in a visual-world paradigm during rapid on-line language processing. Aim 2 (Experiment 3) investigates whether individuals with TBI benefit from observing and producing gesture during word learning. Exploratory Aim 3 examines the relation between speech-gesture integration and working memory abilities to explore individual differences in gesture processing and inform future confirmatory studies. Studying gesture along with speech is critical for providing ecologically valid assessments of language that more closely approximate the real-world communication contexts that characterize and enrich everyday life. The proposed research will directly advance the study of gesture in clinical populations by providing new insight into the ability to integrate speech and gesture in the context of multimodal language processing and testing whether gesture can be leveraged to support new learning in individuals with TBI. This research represents a new direction in TBI research and promises to offer novel and impactful contributions to theories of multimodal communication and to the nature of cognitive-communication deficits after TBI.

项目总结 交流是多模式的，包括语音和手势。当人们交谈时，共同讲话的手势 (手和手臂的自发运动)可以直观地描述言语中传达的信息，但通常 传达不是通过语言传达的独特信息。例如，演讲者可能会说，“我搜索了一个 新食谱“，同时做了一个打字的手势，只传达了说话者在网上搜索的手势 而不是通过一本食谱。听众必须将语言信息与语音和视觉空间信息绑定在一起 从手势到生成消息的集成表示。手势的好处是 在神经典型个体中，沟通和认知都有很好的记录。例如，手势可以提高 对口语信息的理解和记忆，并促进单词学习，这对学术能力至关重要 和事业上的成功。然而，手势在临床人群中还没有得到同样的关注 认知-沟通障碍，如创伤性脑损伤(TBI)。在这份提案中，我们研究了 手势的好处延伸到患有脑外伤的人，或者如果他们的缺陷的本质阻止了手势的 在沟通和认知方面的促进作用。使用一种新的方法，结合了 言语语言病理学、手势研究、心理语言学和神经心理学，我们测试了 患有TBI的个人在多模式语言处理和单词学习过程中使用手势 实验。目标1(实验1和实验2)调查了脑外伤患者的整合能力 在多模式语言处理过程中来自语音和手势的信息。实验一测试的效果 观察叙述者在随后的故事重述中的手势，以确定是否有脑外伤患者 通过手势提供独特的报告信息，并将其集成到他们对整个故事的表示中 时间到了。实验2使用眼球跟踪来确定脑外伤患者是否可以使用手势信息来 在快速在线语言处理期间，解决视觉世界范例中的指代歧义。目标2 (实验3)调查脑外伤患者是否从观察和制作手势中受益 单词学习。探索性目标3考察言语-手势整合与工作的关系 探索手势处理中个体差异的记忆能力，并为未来的验证性研究提供信息。 研究手势和言语对于提供生态上有效的语言评估至关重要， 更接近于描述和丰富日常生活的真实世界交流环境。 这项拟议的研究将直接推动临床人群中手势的研究，为临床人群提供新的 深入了解在多模式语言处理环境中集成语音和手势的能力以及 测试手势是否可以被用来支持脑外伤患者的新学习。这项研究 代表了脑损伤研究的新方向，并有望为理论提供新的和有影响力的贡献 多模式沟通和脑损伤后认知沟通障碍的性质。