Understanding Multi-Modal Discourse: Cognitive Resources and Speech-Gesture Integration

理解多模态话语：认知资源和语音手势整合

• 批准号: 0843946
• 负责人: Seana Coulson
• 金额: $ 30万
• 依托单位: University of California-San Diego
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2009
• 资助国家: 美国
• 起止时间: 2009-08-01 至 2012-07-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=0843946&HistoricalAwards=false
• 关键词: Understanding; Multi; Modal; Discourse; Cognitive

This award is funded under the American Recovery and Reinvestment Act of 2009 (Public Law 111-5.Anyone who has had a phone conversation about how to how to change the oil in a car or how to make lasagna knows the importance of explanatory gestures for communication. Multimodal discourse involves the use of both visual (gestures) and auditory (speech) information during communication. Scientists still have much to learn about how people combine information in gestures with information in speech and whether people vary in their ability to use gestural information. This project evaluates the importance of working memory for combining linguistic information in speech with gestural information. Participant's brain activity will be recorded as they watch videotapes of a person talking about concrete topics such as the shape of objects, their relative sizes, and other spatial information that is difficult to convey through speech alone. The impact of gestural information will be measured by comparing brain activity recorded when the same person watches videos in which the speaker does gesture with those in which he does not. The importance of the different working memory systems will be assessed by seeing how language comprehension suffers when participants are asked to remember irrelevant words (verbal working memory), dot patterns (visuospatial working memory), and pictures of meaningless body positions (sensorimotor working memory). The impact of differences in learning style and cognitive abilities on gesture comprehension will also be examined By discovering the relative importance of verbal, visuospatial, and sensorimotor working memory for understanding face-to-face communication, this research will aid the design of more effective teaching and training methods. The project may guide the development of teaching methods that are specially adapted for people with different learning styles and could help children and adults with communicative deficits by maximizing the effects of gestural information. The project supports an early career scientist and provides summer jobs for college students, including those from minorities under-represented in science.

该奖项根据 2009 年《美国复苏和再投资法案》（公法 111-5）提供资助。任何通过电话交谈过如何更换汽车机油或如何制作烤宽面条的人都知道解释性手势对于交流的重要性。多模态话语涉及在交流过程中使用视觉（手势）和听觉（语音）信息。关于人们如何将手势信息与语音信息结合起来，科学家们还有很多东西需要了解。无论是人 他们使用手势信息的能力各不相同。该项目评估了工作记忆对于将语音中的语言信息与手势信息相结合的重要性。当参与者观看一个人谈论具体主题的录像带时，例如物体的形状、它们的相对大小以及其他难以仅通过语音传达的空间信息，他们的大脑活动将被记录下来。手势信息的影响将通过比较同一个人观看视频时记录的大脑活动来衡量 其中说话者做了手势，而他没有做手势。不同工作记忆系统的重要性将通过观察当参与者被要求记住不相关的单词（言语工作记忆）、点图案（视觉空间工作记忆）和无意义的身体姿势的图片（感觉运动工作记忆）时语言理解能力受到的影响来评估。学习方式和认知能力差异对手势理解的影响也将得到检验 鉴于言语、视觉空间和感觉运动工作记忆对于理解面对面交流的相对重要性，这项研究将有助于设计更有效的教学和培训方法。该项目可以指导开发特别适合不同学习风格的人的教学方法，并可以通过最大限度地发挥手势信息的效果来帮助有沟通缺陷的儿童和成人。该项目支持一名早期职业科学家，并为大学生（包括少数族裔学生）提供暑期工作 在科学领域的代表性不足。