Evaluating the listening effort associated with audiovisual speech

点击翻译按钮获取中文摘要

Project summary One of the most robust findings in the spoken word recognition literature is that listeners can understand speech more successfully when they can see as well as hear the talker. However, there is considerable debate about whether seeing a talker’s face increases or decreases the cognitive and attentional requirements for successfully processing speech (“listening effort”; LE), relative to hearing alone. Conflicting findings in the research may be due, in part, to a lack of systematic evaluation of multiple factors that moderate the effects of visual input on LE. The proposed research, which will involve undergraduate researchers at every step of the process, will be the largest-scale evaluation of this research question to date and will clarify how visual speech information affects LE. The studies will: 1. Compare the LE consequences of adding visual information from a talking face vs an abstract visual stimulus that provides timing but not phonetic information about speech. 2. Assess convergent validity of two commonly used LE measures (a speeded reaction time measure and a memory measure). 3. Determine how difficulty of listening condition (i.e., changes in the level of background noise) affects the LE associated with visual input. 4. Evaluate whether the relationship between visual information and LE differs as a function of the nature of the masking noise. In addition, the work will test whether visual speech information can reduce LE during a more naturalistic listening comprehension task than is typically used in the literature. Given the ubiquity of audiovisual speech and the fact that maintaining high levels of LE can lead to mental fatigue or distress, understanding how visual information affects LE has important practical and clinical consequences. The results of this study will inform the design of devices that generate visual stimuli to accompany spoken language in difficult listening situations, which may be particularly useful for the elderly or hard of hearing.