Multimodal Signaling and Communication Analysis of World Leaders

该项目从多文化、跨学科的角度，使用认知（EC）框架，研究世界领导人之间的多模式沟通。这个框架展示了多模态过程，如话语，语音的声学产生，以及非语言信号，如手势和面部表情，是如何融入人类交流的。欧共体的方法将特别有利于理解不透明环境中的政治，如独裁政权，在那里很难观察到真正的偏好和领导人的优先事项。PI提出的研究代表了计算语言分析，来自音频和视频分析的多模态信号以及生物识别受众数据的新组合。到目前为止，多模态分析还没有全面部署在一个有代表性的国际语料库和现存的研究主要依赖于语料库从英语世界。鉴于国际体系的巨大变化，这种分析为学者们更好地理解国家稳定、冲突过程和政权过渡等问题提供了一种方法。以多模态方法分析国际体系将有助于我们了解更多，而不仅仅是领导人的言论所传达的信息。 该项目通过提供数据集，教学材料和学术出版物来促进对多模态信号重要性的理解，以通过安全和准备计划来翻译多模态输入的含义。文化和语言的差异可能导致误解，进而升级为安全危机。通过评估不同的沟通渠道，PI可以更清楚地了解各国的动机，并更深入地了解它们的偏好和信念。该项目利用生物物理指标和语言，大大提高了科学界对多模式渠道如何与语言协同工作以传达意义的理解，特别是在国际和多元文化背景下。为了完成这项工作，该项目的合作者优先考虑在社会和计算社会科学以及STEM领域雇佣和培训妇女和少数民族。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This project examines multimodal communication among world leaders from a multicultural, interdisciplinary perspective using the Embodied Cognition (EC) framework. This framework demonstrates how multimodal processes, such as discourse, the acoustic production of speech, and nonverbal signals such as gestures and facial expressions, are integrated into human communication. The EC approach will be especially beneficial for understanding politics in opaque environments, like authoritarian regimes, where it is difficult to observe true preferences and priorities of leaders. The PIs' proposed research represents a novel combination of computational linguistic analysis, multimodal signals derived from audio and video analysis, and biometric audience data. To date, multimodal analyses have not been deployed comprehensively on a representative international corpus and extant research relies primarily on corpora from the English-speaking world. Given seismic changes in the international system, this analysis provides a way for scholars to better understand problems like state stability, conflict processes, and regime transition. A multimodal approach to analyzing the international system will help us know more than what leaders’ words alone convey. This project facilitates the understanding of the importance of multimodal signals by providing a data set, pedagogical materials, and academic publications to translate the meaning of multimodal inputs through the Security and Preparedness program. Differences in culture and language may lead to misunderstandings that can escalate to security crises. By evaluating different communication channels, the PIs propose a clearer picture of countries’ motivations and deeper insight into their preferences and beliefs. Using biophysical indicators alongside language, this project substantially improves the scientific community’s understanding of how multimodal channels work in concert with language to convey meaning, especially in international and multicultural contexts. To accomplish this work the collaborators in this project prioritize hiring and training women and minorities in social and computational social sciences, and STEM fields.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.