Understanding Multi-Modal Discourse: Cognitive Resources and Speech-Gesture Integration

该奖项是根据2009年美国复苏和再投资法案（公法111-5）资助的。任何通过电话讨论如何给汽车换油或如何做千层面的人都知道解释性手势对沟通的重要性。多模态语篇包括在交际过程中使用视觉（手势）和听觉（言语）信息。关于人们如何将手势信息与语言信息结合起来，以及人们使用手势信息的能力是否存在差异，科学家们还有很多需要了解的地方。本项目评估工作记忆在结合语言信息和手势信息时的重要性。当参与者观看一个人谈论具体话题的录像带时，比如物体的形状，它们的相对大小，以及其他难以通过言语传达的空间信息，他们的大脑活动将被记录下来。通过比较同一个人观看讲话者做手势和不做手势的视频时所记录的大脑活动，可以测量手势信息的影响。当参与者被要求记住不相关的单词（口头工作记忆）、点图案（视觉空间工作记忆）和无意义的身体姿势图片（感觉运动工作记忆）时，不同工作记忆系统的重要性将通过观察语言理解受到的影响来评估。学习风格和认知能力的差异对手势理解的影响也将被考察。通过发现语言、视觉空间和感觉运动工作记忆对理解面对面交流的相对重要性，本研究将有助于设计更有效的教学和训练方法。该项目可以指导教学方法的发展，特别适合不同学习风格的人，并可以通过最大限度地发挥手势信息的作用来帮助有交流缺陷的儿童和成人。该项目支持一名早期职业科学家，并为大学生提供暑期工作，包括那些在科学领域代表性不足的少数民族学生。

This award is funded under the American Recovery and Reinvestment Act of 2009 (Public Law 111-5.Anyone who has had a phone conversation about how to how to change the oil in a car or how to make lasagna knows the importance of explanatory gestures for communication. Multimodal discourse involves the use of both visual (gestures) and auditory (speech) information during communication. Scientists still have much to learn about how people combine information in gestures with information in speech and whether people vary in their ability to use gestural information. This project evaluates the importance of working memory for combining linguistic information in speech with gestural information. Participant's brain activity will be recorded as they watch videotapes of a person talking about concrete topics such as the shape of objects, their relative sizes, and other spatial information that is difficult to convey through speech alone. The impact of gestural information will be measured by comparing brain activity recorded when the same person watches videos in which the speaker does gesture with those in which he does not. The importance of the different working memory systems will be assessed by seeing how language comprehension suffers when participants are asked to remember irrelevant words (verbal working memory), dot patterns (visuospatial working memory), and pictures of meaningless body positions (sensorimotor working memory). The impact of differences in learning style and cognitive abilities on gesture comprehension will also be examined By discovering the relative importance of verbal, visuospatial, and sensorimotor working memory for understanding face-to-face communication, this research will aid the design of more effective teaching and training methods. The project may guide the development of teaching methods that are specially adapted for people with different learning styles and could help children and adults with communicative deficits by maximizing the effects of gestural information. The project supports an early career scientist and provides summer jobs for college students, including those from minorities under-represented in science.