Evaluating the listening effort associated with audiovisual speech

项目总结 口语识别文献中最有说服力的发现之一是，听众可以理解 当他们既能看到也能听到说话者时，他们的演讲就会更成功。然而，有相当大的争议。 关于看到说话者的脸是增加还是减少了认知和注意力要求 成功地处理语音(“听力努力”；LE)，相对于单纯的听力。研究中相互矛盾的发现 研究可能在一定程度上是由于缺乏对多种因素的系统评估，这些因素缓和了 LE上的视觉输入。这项拟议的研究将涉及本科生研究人员在研究的每一步 过程，将是迄今为止对这一研究问题进行的最大规模的评估，并将阐明视觉语音 信息影响LE。这些研究将会： 1.比较从说话的脸部添加视觉信息与从抽象的视觉添加视觉信息的LE结果 提供时序信息而不是语音信息的刺激。 2.评估两种常用的LE度量的收敛效度(加速反应时度量和 记忆测量)。 3.确定听力条件的难度(即背景噪声水平的变化)对听力的影响 与视觉输入关联的LE。 4.评估视觉信息和LE之间的关系是否作为性质的函数而不同 掩蔽的噪音。 此外，这项工作将测试视觉语音信息是否可以在更自然的过程中减少LE 听力理解任务比文献中通常使用的要多。鉴于视听演讲无处不在 事实上，保持高水平的LE会导致精神疲劳或痛苦，了解视觉 信息影响LE具有重要的现实意义和临床意义。这项研究的结果将告知 在困难的听力环境中产生视觉刺激以伴随口语的设备的设计， 这可能对老年人或重听人特别有用。

Project summary One of the most robust findings in the spoken word recognition literature is that listeners can understand speech more successfully when they can see as well as hear the talker. However, there is considerable debate about whether seeing a talker’s face increases or decreases the cognitive and attentional requirements for successfully processing speech (“listening effort”; LE), relative to hearing alone. Conflicting findings in the research may be due, in part, to a lack of systematic evaluation of multiple factors that moderate the effects of visual input on LE. The proposed research, which will involve undergraduate researchers at every step of the process, will be the largest-scale evaluation of this research question to date and will clarify how visual speech information affects LE. The studies will: 1. Compare the LE consequences of adding visual information from a talking face vs an abstract visual stimulus that provides timing but not phonetic information about speech. 2. Assess convergent validity of two commonly used LE measures (a speeded reaction time measure and a memory measure). 3. Determine how difficulty of listening condition (i.e., changes in the level of background noise) affects the LE associated with visual input. 4. Evaluate whether the relationship between visual information and LE differs as a function of the nature of the masking noise. In addition, the work will test whether visual speech information can reduce LE during a more naturalistic listening comprehension task than is typically used in the literature. Given the ubiquity of audiovisual speech and the fact that maintaining high levels of LE can lead to mental fatigue or distress, understanding how visual information affects LE has important practical and clinical consequences. The results of this study will inform the design of devices that generate visual stimuli to accompany spoken language in difficult listening situations, which may be particularly useful for the elderly or hard of hearing.