Evaluating the listening effort associated with audiovisual speech

项目摘要 口语单词识别文献中最有力的发现之一是， 当他们既能看到说话者，又能听到说话者时，他们的演讲会更成功。然而， 关于看到说话者的脸是增加还是减少了对认知和注意力的要求， 成功地处理语音（“听力努力”; LE），相对于单独的听力。调查结果在 研究可能部分是由于缺乏对多种因素的系统评估，这些因素会减轻 LE上的视觉输入。拟议的研究，这将涉及本科研究人员在每一步的 过程中，将是最大规模的评估这一研究问题的日期，并将澄清如何视觉语音 信息影响LE。这些研究将： 1.比较从一个说话的脸和一个抽象的视觉信息中添加视觉信息的LE后果。 提供时间但不提供语音信息的刺激。 2.评估两种常用LE测量（快速反应时间测量和 记忆测量）。 3.确定听力条件的难度（即，背景噪声水平的变化）影响 与视觉输入相关的LE。 4.评估视觉信息和LE之间的关系是否因性质而异 屏蔽噪音。 此外，这项工作将测试视觉语音信息是否可以减少LE在一个更自然的 听力理解任务比通常使用的文献。鉴于视听语言的普遍性 事实上，保持高水平的LE会导致精神疲劳或痛苦，了解视觉如何 信息影响LE具有重要的实际和临床后果。这项研究的结果将告知 在困难的听力情况下，设计产生视觉刺激的设备来伴随口语， 这对于老年人或听力障碍者可能特别有用。

Project summary One of the most robust findings in the spoken word recognition literature is that listeners can understand speech more successfully when they can see as well as hear the talker. However, there is considerable debate about whether seeing a talker’s face increases or decreases the cognitive and attentional requirements for successfully processing speech (“listening effort”; LE), relative to hearing alone. Conflicting findings in the research may be due, in part, to a lack of systematic evaluation of multiple factors that moderate the effects of visual input on LE. The proposed research, which will involve undergraduate researchers at every step of the process, will be the largest-scale evaluation of this research question to date and will clarify how visual speech information affects LE. The studies will: 1. Compare the LE consequences of adding visual information from a talking face vs an abstract visual stimulus that provides timing but not phonetic information about speech. 2. Assess convergent validity of two commonly used LE measures (a speeded reaction time measure and a memory measure). 3. Determine how difficulty of listening condition (i.e., changes in the level of background noise) affects the LE associated with visual input. 4. Evaluate whether the relationship between visual information and LE differs as a function of the nature of the masking noise. In addition, the work will test whether visual speech information can reduce LE during a more naturalistic listening comprehension task than is typically used in the literature. Given the ubiquity of audiovisual speech and the fact that maintaining high levels of LE can lead to mental fatigue or distress, understanding how visual information affects LE has important practical and clinical consequences. The results of this study will inform the design of devices that generate visual stimuli to accompany spoken language in difficult listening situations, which may be particularly useful for the elderly or hard of hearing.