RUI: Effects of visual phonetic similarity on audiovisual spoken word recognition

口语单词识别被定义为将声音模式与“心理词典”中的含义进行匹配的过程--“心理词典”是成人语言使用者所知道的大约85000个单词的记忆库。以前的研究已经证明，在听觉上与给定单词相似的词条数量会影响该单词被识别的容易程度。然而，语音识别不仅仅是一种声学现象；观看某人说话还可以提供有关话语内容的额外信息。虽然目前对声学相似性在口语单词识别中的作用有了坚实的理解，但对视觉相似性知之甚少，而且当两种信息源都可用时，关于声学和视觉相似性的相互作用也知之甚少。这可能是因为视觉相似性特别难以衡量。一个问题是，声学上不同的语音单元在视觉上可能是相同的。视觉上相同的单词被认为构成了词汇等效类(LEC)。先前的研究表明，驻留在LEC中的单词数量会影响这些单词的口述容易程度。同样，LEC的大小影响视觉信息增强听觉语音识别的程度。然而，LEC的构成在某种程度上取决于说话人。这项研究将实现两个目标：第一，将建立一个可公开访问的计算工具，以促进视觉词汇相似性的计算和实验研究。其次，几个行为实验将进一步阐明视觉相似性在口语识别中所起的作用。这个项目将促进我们对口语识别所涉及的基本机制的理解。这些知识将对处理聋人或听力受损人群的临床医生和致力于自动语音识别问题的工程师有用。

Spoken word recognition is defined as the process by which acoustic patterns are matched to meanings in the "mental lexicon" -- the memory repository of the approximately 85000 words known by an adult language speaker. Previous research has demonstrated that the number of lexical entries acoustically similar to a given word influences the ease with which that word is recognized. However, speech recognition is not solely an acoustic phenomenon; watching someone speak also provides additional information about the content of an utterance. Although there is presently a solid understanding of the role of acoustic similarity in spoken word recognition, less is known about visual similarity, and very little is known about the interaction of acoustic and visual similarity when both sources of information are available. This may be because visual similarity is particularly difficult to measure. One problem is that speech units that are acoustically different can be visually identical. Words that are visually identical are said to comprise a Lexical Equivalence Class (LEC). Previous research has shown that the number of words residing in an LEC affects the ease with which those words are lipread. Similarly, the size of an LEC affects the extent to which visual information enhances the recognition of auditory speech. However, the makeup of an LEC is to some extent dependent on the speaker. The proposed research will accomplish two goals: first, a publicly accessible computational tool will be built to facilitate computational and experimental investigations of visual lexical similarity. Second, several behavioral experiments will further elucidate the role that visual similarity plays in spoken word recognition. This project will advance our understanding of the basic mechanisms involved in spoken word recognition. Such knowledge will be useful for clinicians working with deaf or hearing-impaired populations, and engineers working on problems in automatic speech recognition.