World Futures: Multimodal Viewpoint Construction by Russian International Media

在世界正在与COVID-19作斗争之际，对未来的讨论从未像现在这样重要和具有挑战性。人们参与讨论大流行后我们的生活将会是什么样子，媒体也是如此。媒体经常谈论期货来框定我们的思维方式，俄罗斯国家资助的媒体利用植根于期货描述的观点构建，作为操纵和影响公众舆论的一种微妙但有力的方法。视点构建技术不仅是口头的，而且是多模态的：它们结合了单词、韵律线索（如语调和时间）、手势和运动以及其他非口头元素。它们严重依赖文化、历史和其他背景知识，如果被用于操纵目的，就会导致观众得出错误的结论，做出评估，或得出有利于敌对行为者的意见，而没有意识到这正在发生。在数字人文学科的真正精神中，我们为期两年的项目将融合语言、韵律和手势的认知和基于语料库的分析与区域研究（区域知识：文化、文学、历史和社会），同时利用机器学习、自然语言处理（NLP）和计算机视觉（CV）的最新发展来大规模解决问题。我们将把我们对俄罗斯国际媒体广播（英语和俄语）的多模态分析与受众在社交媒体上的评论分析结合起来，打开一扇窗口，了解受众心中的观点构建，从而为我们的语言分析提供额外的有效性。我们的项目将在认知语言学和语料库语言学领域开辟新路，显著提高目前语言学以及其他人文社会科学中使用的大媒体数据分析的整体可靠性。我们的方法将有助于回答学术研究人员和西方政策制定者提出的有关未来信息威胁和缓解的问题，并将其应用于俄罗斯这个具有战略重要性的地区。最终，该项目将超越应用语言学方法来分析虚假信息。它将使研究人员能够以系统的方式测试发现的多模态模式，从而解决语言学家面临的最大挑战之一。此外，该项目将有助于回答理论语言学（语义学、句法学、语用学、比较学）中多模态研究中的一些重大问题，如：1)多模态意义是如何构建的，多模态意义是如何被感知的？2)在手势、韵律和言语模式中，意义的建构是如何分布的？手势和韵律的“冗余”模式真的冗余吗？3)根植于不同历史文化背景的语法和语音特征截然不同的语言，如何在追求相同的总体交际和语用目标的同时，多模态地实现时间和空间的概念？

Discussions of futures have never been more important and challenging than now when the world finds itself fighting COVID-19. People engage in conversations about what our lives will look like after the pandemic, and so do the media.The media often talk about futures to frame the way we think, and the media funded by the Russian State use viewpoint construction rooted in the depiction of futures as a subtle but powerful approach to manipulate and influence public opinion.Viewpoint construction techniques are not just verbal but multimodal: they combine words, prosodic cues such as intonation and timing, gesture, and movement, and other non-verbal elements. They rely heavily on culture, history, and other contextual knowledge, and if used for manipulative purposes, lead the viewer to draw a false conclusion, make an assessment, or come to an opinion that is beneficial to the hostile actor, without being aware that this is happening.In the true spirit of the digital humanities, our 2-year project will fuse cognitive and corpus-driven/based analyses of language, prosody, and gesture with area studies (regional knowledge: culture, literature, history, and society), while leveraging latest developments in machine learning, natural language processing (NLP) and computer vision (CV) to tackle the problem at scale.We will combine our multimodal analysis of Russian international media broadcasts (English and Russian) with the analysis of audiences' comments on social media to open a window onto viewpoint construction in the audiences' minds and thus provide additional validity to our linguistic analysis. Our project will break new ground in cognitive and corpus linguistics to significantly increase the overall reliability of large media data analysis currently used in linguistics as well as other humanities and the social sciences. Our approach will help answer questions posed by academic researchers and Western policymakers in relation to future information threats and mitigations, as applied to Russia as a region of strategic importance. Ultimately, the project will go beyond the applied linguistics approach to disinformation analysis. It will enable researchers to test multimodal patterns found in a systematic way, thereby addressing one of the biggest challenges linguists face. Furthermore, the project will contribute to answering such big questions in multimodal research in theoretical linguistics (semantics, syntax, pragmatics, comparative) as:1) How are meanings constructed multimodally and how are multimodal meanings perceived?2) How is the construction of meanings distributed across gestural, prosodic, and verbal modes? Are the "redundant" modes of gesture and prosody really redundant?3) How do languages with quite distinct grammatical and phonetic properties, rooted in quite diverse historical and cultural backgrounds, realise the ideas of time and space multimodally while pursuing the same overarching communicative and pragmatic goals?