When words speak off the page: Covert emotional prosodic processing in silent reading of direct quotations

当文字脱离页面时：默读直接引文中隐秘的情感韵律处理

• 批准号: ES/N002784/1
• 负责人: Bo Yao
• 金额: $ 27.6万
• 依托单位: University of Manchester
• 依托单位国家: 英国
• 项目类别: Research Grant
• 财政年份: 2016
• 资助国家: 英国
• 起止时间: 2016 至 无数据
• 项目状态: 已结题
• 来源: https://gtr.ukri.org/projects?ref=ES%2FN002784%2F1
• 关键词: When; words; speak; off; page

Written communication (e.g., emails, news reports, social media) is a major form of social information exchange in today's world. However, it is sometimes difficult to interpret the intended meaning of a written message without hearing prosody (rhythm, stress, and intonation of speech) that is instrumental in understanding the writer's feelings, attitudes, and intentions. For example, a prosody-less "thank you" email can be confusing as to whether the sender is being sincere or sarcastic (Kruger et al., 2005). Emails like these are often misinterpreted as being more negative or neutral than intended; such miscommunications can damage social cohesiveness and group identity within organisations and communities, thereby undermining economic performance and societal stability (Byron, 2008).Interestingly, written words may not be entirely "silent" after all. My recent research showed that we mentally (or covertly) simulate speech prosody (or "inner voices") during silent reading of written direct quotations (Mary gasped: "This dress is beautiful!") as if we were hearing someone speaking (Yao et al., 2011, 2012). For example, Yao and colleagues (2011) observed that silent reading of direct quotations elicited higher neural activity in voice-selective areas of the auditory cortex as compared to silent reading of meaning-equivalent indirect speech (Mary gasped that the dress was beautiful.).Can such covert prosody compensate for the lack of overt speech prosody in written language and thus enhance written communication? To address this question, the proposed project will systematically examine the nature (is covert prosody sound- or action-based in nature?), mechanisms (what information processing systems are engaged?) and emotional consequences (does covert prosody induce emotions and thereby influence behaviour?) of covert prosodic processing in silent reading of written direct quotations.Theoretically motivated by the working neural models for "overt" emotional prosodic processing in speech (e.g., Schirmer & Kotz, 2006), the current proposal will probe "where" and "when" in the brain covert prosodic cues of various natures are mentally simulated and integrated into coherent covert prosodic representations and how these representations consequently induce emotional responses and aid in inferring the quoted speaker's mental state. Using complementary neuroimaging techniques, it will localise the neural substrates of systems engaged in covert emotional prosodic processing (fMRI), specify the time courses of the information processes within these systems (EEG, MEG), and integrate this information to form a unified spatio-temporal neural model for covert emotional prosodic processing.The findings of this project have clear implications for the theoretical development of emotional prosody-based social communication, embodied cognition, and speech pragmatics, and will be of interest to all written language users (e.g., communication-based enterprises, social services, and the wider public). This research also has potential impact on early language education and diagnosis of Parkinson's disease (PD). For example, understanding direct quotations requires the reader to take the quoted speaker's perspective and attribute emotions and mental states to them. A quotation-rich teaching method thus may effectively enhance children's Theory of Mind ability (ability to attribute mental states) that is crucial in their cognitive development and social cognition. Moreover, PD patients may struggle in simulating covert emotional prosody due to their motor (articulation) dysfunction. Consequently, they may display difficulty in understanding figurative speech quotations (e.g., they may not detect the sarcasm in - She rolled her eyes, grumbling: "What a sunny day!"). This research could thus motivate the development of a low-cost quotation-based diagnostic tool for monitoring PD progression.

书面沟通（例如，电子邮件、新闻报道、社交媒体）是当今世界社会信息交换的主要形式。然而，有时很难解释一个书面信息的意图，而不听韵律（节奏，重音和语音语调），这是在理解作者的感情，态度和意图的工具。例如，一封没有韵律的“谢谢你”电子邮件可能会让人搞不清发件人是真诚的还是讽刺的（Kruger等人，2005年）。像这样的电子邮件往往被误解为比预期的更消极或中立;这种错误的沟通可能会破坏组织和社区内的社会凝聚力和群体认同感，从而破坏经济表现和社会稳定（Byron，2008）有趣的是，书面文字可能并不完全是“沉默”的。我最近的研究表明，我们在无声阅读书面直接引文时，会在精神上（或秘密地）模拟言语韵律（或“内心声音”）（玛丽气喘吁吁地说：“这件衣服太漂亮了！“）就好像我们听到有人在说话（Yao等人，2011年、2012年）。例如，Yao及其同事（2011）观察到，与无声阅读意义等同的间接引语相比，无声阅读直接引语会引发听觉皮层中声音选择区域的更高神经活动（玛丽惊叹于这件衣服很漂亮）。这种隐性韵律是否可以弥补书面语中显性语音韵律的不足，从而增强书面语的交际效果？为了解决这个问题，拟议的项目将系统地研究性质（是隐蔽韵律声音或行动的性质？），机制（使用哪些信息处理系统？）和情绪后果（内隐韵律是否会诱发情绪，从而影响行为？）在对书面直接引语的无声阅读中的内隐韵律处理的研究中，理论上受到言语中“外显”情感韵律处理的神经模型的激励（例如，Schirmer & Kotz，2006），目前的建议将探讨“在哪里”和“什么时候”在大脑中的各种性质的隐性韵律线索的心理模拟和整合成连贯的隐性韵律表征，以及这些表征如何因此诱导情绪反应，并帮助推断被引述的扬声器的心理状态。利用互补的神经成像技术，它将定位参与内隐情绪韵律加工（fMRI）的系统的神经基质，确定这些系统内信息加工的时间进程（脑电图，脑磁图），并整合这些信息，形成一个统一的情绪韵律内隐加工的时空神经模型。本研究的发现对情绪韵律的理论发展具有明确的意义--基于社会交际，具身认知和言语语用学，并且将是所有书面语言用户感兴趣的（例如，企业、社会服务和更广泛的公众）。本研究对早期语言教育和帕金森病的诊断也有潜在的影响。例如，理解直接引语需要读者从被引用者的角度出发，并将情感和精神状态归因于他们。因此，丰富的教学方法可以有效地提高儿童的心理理论能力（归因心理状态的能力），这是他们的认知发展和社会认知的关键。此外，PD患者可能会因为运动（发音）功能障碍而难以模拟内隐情感韵律。因此，他们可能会在理解比喻性话语引用方面表现出困难（例如，她翻了翻白眼，抱怨道：“多么晴朗的一天啊！").因此，这项研究可以促进开发一种低成本的基于报价的诊断工具，用于监测PD进展。

项目成果

Direct speech quotations promote low relative-clause attachment in silent reading of English.

直接引语可以促进英语默读中较低的关系从句依恋。

• DOI: 10.1016/j.cognition.2018.03.017
• 发表时间: 2018
• 期刊: Cognition
• 影响因子: 3.4
• 作者: Yao B

Reading direct speech quotes increases theta phase-locking: Evidence for cortical tracking of inner speech?

• DOI: 10.1016/j.neuroimage.2021.118313
• 发表时间: 2021-06
• 期刊: NeuroImage
• 影响因子: 5.7
• 作者: Bo Yao;Jason R. Taylor;B. Banks;S. Kotz

What can size tell us about abstract conceptual processing?

关于抽象概念处理，大小可以告诉我们什么？

• DOI: 10.31234/osf.io/7dnye
• 发表时间: 2021
• 作者: Yao B

Reading direct speech quotes increases theta phase-locking evidence for cortical tracking of inner speech?

阅读直接言语引用会增加大脑皮层跟踪内部言语的 θ 锁相证据？

• DOI: 10.31234/osf.io/vta7d
• 发表时间: 2020
• 作者: Yao B