Multimodal Signaling and Communication Analysis of World Leaders

本项目使用具身认知（EC）框架，从多元文化、跨学科的角度审视世界领导人之间的多模式沟通。该框架展示了多模态过程，如话语，语音的声学产生，以及非语言信号，如手势和面部表情，是如何整合到人类交流中的。欧共体的方法对于理解不透明环境中的政治尤其有益，比如威权政权，在那里很难观察到领导人的真正偏好和优先事项。pi提出的研究代表了计算语言分析、来自音频和视频分析的多模态信号以及生物识别受众数据的新组合。迄今为止，多模态分析还没有在具有代表性的国际语料库上得到全面的应用，现有的研究主要依赖于英语世界的语料库。鉴于国际体系发生了翻天覆地的变化，这种分析为学者们更好地理解国家稳定、冲突过程和政权过渡等问题提供了一种途径。多模式分析国际体系的方法将帮助我们了解更多，而不仅仅是领导人的话语所传达的信息。该项目通过提供数据集、教学材料和学术出版物，通过安全和准备项目翻译多模式输入的含义，促进对多模式信号重要性的理解。文化和语言的差异可能导致误解，进而升级为安全危机。通过评估不同的沟通渠道，ppi更清晰地描绘了各国的动机，并更深入地了解了他们的偏好和信念。该项目使用生物物理指标和语言，大大提高了科学界对多模式渠道如何与语言协同工作以传达意义的理解，特别是在国际和多元文化背景下。为了完成这项工作，该项目的合作者优先考虑在社会科学和计算社会科学以及STEM领域雇用和培训妇女和少数民族。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project examines multimodal communication among world leaders from a multicultural, interdisciplinary perspective using the Embodied Cognition (EC) framework. This framework demonstrates how multimodal processes, such as discourse, the acoustic production of speech, and nonverbal signals such as gestures and facial expressions, are integrated into human communication. The EC approach will be especially beneficial for understanding politics in opaque environments, like authoritarian regimes, where it is difficult to observe true preferences and priorities of leaders. The PIs' proposed research represents a novel combination of computational linguistic analysis, multimodal signals derived from audio and video analysis, and biometric audience data. To date, multimodal analyses have not been deployed comprehensively on a representative international corpus and extant research relies primarily on corpora from the English-speaking world. Given seismic changes in the international system, this analysis provides a way for scholars to better understand problems like state stability, conflict processes, and regime transition. A multimodal approach to analyzing the international system will help us know more than what leaders’ words alone convey. This project facilitates the understanding of the importance of multimodal signals by providing a data set, pedagogical materials, and academic publications to translate the meaning of multimodal inputs through the Security and Preparedness program. Differences in culture and language may lead to misunderstandings that can escalate to security crises. By evaluating different communication channels, the PIs propose a clearer picture of countries’ motivations and deeper insight into their preferences and beliefs. Using biophysical indicators alongside language, this project substantially improves the scientific community’s understanding of how multimodal channels work in concert with language to convey meaning, especially in international and multicultural contexts. To accomplish this work the collaborators in this project prioritize hiring and training women and minorities in social and computational social sciences, and STEM fields.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.