RUI: Effects of visual phonetic similarity on audiovisual spoken word recognition

口语单词识别被定义为将声音模式与“心理词典”中的含义相匹配的过程，心理词典是一个成年人所知道的大约85000个单词的记忆库。先前的研究已经表明，与给定单词声学相似的词条数量影响该单词被识别的难易程度。然而，语音识别不仅仅是一种声学现象;观看某人说话还提供了有关话语内容的附加信息。虽然目前有一个坚实的理解的作用，声学相似性在口语单词识别，较少的是已知的视觉相似性，很少有人知道的交互作用的声学和视觉相似性时，这两个来源的信息是可用的。 这可能是因为视觉相似性特别难以测量。一个问题是，声音上不同的语音单元在视觉上可能是相同的。视觉上相同的单词被称为词汇等价类（LEC）。先前的研究表明，LEC中的单词数量会影响这些单词的lipread的容易程度。类似地，LEC的大小影响视觉信息增强听觉语音识别的程度。然而，LEC的构成在某种程度上取决于扬声器。本研究的目的有二：第一，建立一个可公开存取的计算工具，以促进视觉词汇相似性的计算与实验研究。其次，一些行为实验将进一步阐明视觉相似性在口语单词识别中的作用。这个项目将促进我们对口语单词识别的基本机制的理解。这些知识将对聋人或听力受损人群的临床医生以及研究自动语音识别问题的工程师有用。

Spoken word recognition is defined as the process by which acoustic patterns are matched to meanings in the "mental lexicon" -- the memory repository of the approximately 85000 words known by an adult language speaker. Previous research has demonstrated that the number of lexical entries acoustically similar to a given word influences the ease with which that word is recognized. However, speech recognition is not solely an acoustic phenomenon; watching someone speak also provides additional information about the content of an utterance. Although there is presently a solid understanding of the role of acoustic similarity in spoken word recognition, less is known about visual similarity, and very little is known about the interaction of acoustic and visual similarity when both sources of information are available. This may be because visual similarity is particularly difficult to measure. One problem is that speech units that are acoustically different can be visually identical. Words that are visually identical are said to comprise a Lexical Equivalence Class (LEC). Previous research has shown that the number of words residing in an LEC affects the ease with which those words are lipread. Similarly, the size of an LEC affects the extent to which visual information enhances the recognition of auditory speech. However, the makeup of an LEC is to some extent dependent on the speaker. The proposed research will accomplish two goals: first, a publicly accessible computational tool will be built to facilitate computational and experimental investigations of visual lexical similarity. Second, several behavioral experiments will further elucidate the role that visual similarity plays in spoken word recognition. This project will advance our understanding of the basic mechanisms involved in spoken word recognition. Such knowledge will be useful for clinicians working with deaf or hearing-impaired populations, and engineers working on problems in automatic speech recognition.