The use of visible speech cues for improving auditory detection of spoken sentences
The use of visible speech cues for improving auditory detection of spoken sentences
复制标题
DOI：
10.1121/1.1288668
复制
发表时间：
2000-09-01
影响因子：
2.4
通讯作者：
Seitz, PF
中科院分区：
文献类型：
--
作者：
Grant, KW;Seitz, PF
经典的演讲阅读对语音识别益处的描述将听觉和视觉通道视为独立的信息源，在语音感知过程中相当早地整合在一起。这项研究的主要问题是，是否可以利用语音发音臂的可见运动来提高对噪声中语音的检测，从而表明朗读对语音检测能力的影响，而不是识别语音的能力。在第一个实验中，10名听力正常的受试者在三种条件下检测了三个已知口语句子的存在：纯听觉(A)、听觉加语音阅读和视觉匹配句子(AV(M))和听觉加语音阅读(AV(UM))。当言语阅读句子与目标句子匹配时，平均检测阈值相对于听觉条件提高了约1.6dB。然而，三个目标句的阈值降低量有显著差异(从0.8分贝到2.2分贝)。AV(UM)状态和A状态之间的检测阈值没有差异。在第二个实验中，我们考察了视觉匹配的正字法刺激对相同三个目标句子的检测阈值的影响，这六个被试都参与了之前的实验。当在每次试验前呈现正交刺激时，平均检测阈值相对于A条件提高了约0.5分贝。然而，与AV(M)条件不同的是，正字法导致的检测改进不依赖于目标句子。对从每个句子的选定频谱区域(对应于宽带语音，以及第一、第二和第三共振峰区域)得到的声包络与张嘴面积之间的相关性分析表明，AV(M)阈值的降低可能取决于听觉-视觉时间一致性的程度，特别是嘴唇张开面积与来自中、高频声能的包络之间的相关性。综上所述，这些数据(至少对于这些句子)表明，从言语产生过程中事实的动态运动中获得的视觉线索与时间对齐的听觉线索相互作用，以提高听觉检测的灵敏度。视觉影响的大小在一定程度上取决于声学包络和发音关节的可视运动之间的相关程度。[S0001-4966(00)03709-7]。
Classic accounts of the benefits of speechreading to speech recognition treat auditory and visual channels as independent sources of information that are integrated fairly early in the speech perception process. The primary question addressed in this study was whether visible movements of the speech articulators could be used to improve the detection of speech in noise, thus demonstrating an influence of speechreading on the ability to detect, rather than recognize, speech. In the first experiment, ten normal-hearing subjects detected the presence of three known spoken sentences in noise under three conditions: auditory-only (A), auditory plus speechreading with a visually matched sentence (AV(M)) and auditory plus speechreading with a visually unmatched sentence (AV(UM)). When the speechread sentence matched the target sentence, average detection thresholds improved by about 1.6 dB relative to the auditory condition. However, the amount of threshold reduction varied significantly for the three target sentences (from 0.8 to 2.2 dB). There was no difference in detection thresholds between the AV(UM) condition and the A condition. In a second experiment, the effects of visually matched orthographic stimuli on detection thresholds was examined for the same three target sentences in six subjects who participated in the earlier experiment. When the orthographic stimuli were presented just prior to each trial, average detection thresholds improved by about 0.5 dB relative to the A condition. However, unlike the AV(M) condition, the detection improvement due to orthography was not dependent on the target sentence. Analyses of correlations between area of mouth opening and acoustic envelopes derived from selected spectral regions of each sentence (corresponding to the wide-band speech, and first, second, and third formant regions) suggested that AV(M) threshold reduction may be determined by the degree of auditory-visual temporal coherence, especially between the area of lip opening and the envelope derived from mid- to high-frequency acoustic energy. Taken together, the data (for these sentences at least) suggest that visual cues derived from the dynamic movements of the fact during speech production interact with time-aligned auditory cues to enhance sensitivity in auditory detection. The amount of visual influence depends in part on the degree of correlation between acoustic envelopes and visible movement of the articulators. [S0001-4966(00)03709-7].