When words speak off the page: Covert emotional prosodic processing in silent reading of direct quotations

书面交流(如电子邮件、新闻报道、社交媒体)是当今世界社会信息交流的主要形式。然而，如果听不到韵律(节奏、重音和语调)，就很难理解书面信息的意图，因为韵律有助于理解作者的感受、态度和意图。例如，一封没有韵律的“谢谢”电子邮件可能会混淆发送者是真诚的还是讽刺的(Kruger等人，2005)。这样的电子邮件经常被误解为比原意更负面或更中立；这种沟通失误会损害组织和社区内的社会凝聚力和群体认同感，从而破坏经济表现和社会稳定(拜伦，2008)。有趣的是，书面语言可能终究不会完全“沉默”。我最近的研究表明，在默读直接引用的文字时，我们会在心里(或偷偷地)模拟语音韵律(或“内心的声音”)(玛丽气喘吁吁地说：“这件衣服太美了！”)就像我们听到有人在说话一样(姚等人，2011,2012)。例如，姚和他的同事(2011)观察到，与默读意义相同的间接引语相比，默读直接引语会在听觉皮质的语音选择区域引发更高的神经活动(玛丽惊叹这件衣服很漂亮。)这种隐蔽的韵律能弥补书面语言中缺乏公开的语音韵律，从而加强书面交流吗？为了解决这个问题，拟议的项目将系统地检查性质(本质上是隐蔽的韵律声音还是基于动作？)、机制(使用什么信息处理系统？)以及情绪后果(隐蔽的韵律是否会引发情绪，从而影响行为？)在言语中“显性”情感韵律加工的工作神经模型(如Schirmer&Kotz，2006)的理论基础上，本研究将探讨大脑中不同性质的隐性韵律线索在大脑中的“何时”和“何处”被心理模拟并整合到连贯的隐性韵律表征中，以及这些表征是如何诱导情绪反应并帮助推断被引用者的心理状态的。利用互补的神经成像技术，它将定位参与隐性情绪韵律处理(FMRI)的系统的神经基础，指定这些系统(EEG，MEG)中信息处理的时间进程，并整合这些信息，形成用于隐性情绪韵律处理的统一的时空神经模型。该项目的发现对基于情绪韵律的社交沟通、体现认知和言语语用的理论发展具有明确的启示，并将引起所有书面语言用户(例如，基于沟通的企业、社会服务机构和更广泛的公众)的兴趣。这项研究对帕金森病的早期语言教育和诊断也有潜在的影响。例如，理解直接引语需要读者接受被引述人的观点，并将情绪和心理状态归因于他们。因此，引语丰富的教学方法可以有效地提高儿童的心理理论能力(归因于心理状态的能力)，这对他们的认知发展和社会认知是至关重要的。此外，由于运动(发音)功能障碍，帕金森病患者可能难以模拟隐蔽的情绪韵律。因此，他们可能会表现出理解比喻引语的困难(例如，他们可能没有察觉到其中的讽刺--她翻了翻眼睛，抱怨道：“多么晴朗的一天啊！”)。因此，这项研究可以推动开发一种低成本的基于报价的诊断工具来监测PD的进展。

Written communication (e.g., emails, news reports, social media) is a major form of social information exchange in today's world. However, it is sometimes difficult to interpret the intended meaning of a written message without hearing prosody (rhythm, stress, and intonation of speech) that is instrumental in understanding the writer's feelings, attitudes, and intentions. For example, a prosody-less "thank you" email can be confusing as to whether the sender is being sincere or sarcastic (Kruger et al., 2005). Emails like these are often misinterpreted as being more negative or neutral than intended; such miscommunications can damage social cohesiveness and group identity within organisations and communities, thereby undermining economic performance and societal stability (Byron, 2008).Interestingly, written words may not be entirely "silent" after all. My recent research showed that we mentally (or covertly) simulate speech prosody (or "inner voices") during silent reading of written direct quotations (Mary gasped: "This dress is beautiful!") as if we were hearing someone speaking (Yao et al., 2011, 2012). For example, Yao and colleagues (2011) observed that silent reading of direct quotations elicited higher neural activity in voice-selective areas of the auditory cortex as compared to silent reading of meaning-equivalent indirect speech (Mary gasped that the dress was beautiful.).Can such covert prosody compensate for the lack of overt speech prosody in written language and thus enhance written communication? To address this question, the proposed project will systematically examine the nature (is covert prosody sound- or action-based in nature?), mechanisms (what information processing systems are engaged?) and emotional consequences (does covert prosody induce emotions and thereby influence behaviour?) of covert prosodic processing in silent reading of written direct quotations.Theoretically motivated by the working neural models for "overt" emotional prosodic processing in speech (e.g., Schirmer & Kotz, 2006), the current proposal will probe "where" and "when" in the brain covert prosodic cues of various natures are mentally simulated and integrated into coherent covert prosodic representations and how these representations consequently induce emotional responses and aid in inferring the quoted speaker's mental state. Using complementary neuroimaging techniques, it will localise the neural substrates of systems engaged in covert emotional prosodic processing (fMRI), specify the time courses of the information processes within these systems (EEG, MEG), and integrate this information to form a unified spatio-temporal neural model for covert emotional prosodic processing.The findings of this project have clear implications for the theoretical development of emotional prosody-based social communication, embodied cognition, and speech pragmatics, and will be of interest to all written language users (e.g., communication-based enterprises, social services, and the wider public). This research also has potential impact on early language education and diagnosis of Parkinson's disease (PD). For example, understanding direct quotations requires the reader to take the quoted speaker's perspective and attribute emotions and mental states to them. A quotation-rich teaching method thus may effectively enhance children's Theory of Mind ability (ability to attribute mental states) that is crucial in their cognitive development and social cognition. Moreover, PD patients may struggle in simulating covert emotional prosody due to their motor (articulation) dysfunction. Consequently, they may display difficulty in understanding figurative speech quotations (e.g., they may not detect the sarcasm in - She rolled her eyes, grumbling: "What a sunny day!"). This research could thus motivate the development of a low-cost quotation-based diagnostic tool for monitoring PD progression.