CAREER: Integrating Perceptual and Linguistic Information in Models of Semantic Representation

人类从语言和感知信息的重复经验中学习单词和更大的话语单位的意义。然而，目前的语义学习和表示计算模型只关注语言结构。这个职业奖探索人类如何使用多感官知觉和语言经验来组织语义记忆。印第安纳大学的迈克尔·琼斯博士将使用专门设计的基于网络的实验方案来收集大量关于词语指代物感知结构的数据。例如，给参与者一个目标单词，要求他们列出一系列描述这个单词的语言属性（例如，给定一个狗，参与者可能会说出“有四条腿，有皮毛，吠叫”等）。然后，第二个参与者只被提供了属性列表，并被要求猜测被描述的是哪个单词。或者，向第二个参与者提供的信息可以是向第一个参与者提供的目标单词表示的内容的绘图，第一个参与者已将预定义的对象组件配置为2D或3D显示。通过将这些实验设计为基于互联网的，琼斯博士将收集大量数据来设计和测试单词学习过程中语言和感知整合的计算机模型。这项研究产生的模型可以更好地理解以语言缺陷为特征的学习障碍（例如许多形式的自闭症）和以语义紊乱为特征的与年龄相关的障碍（例如阿尔茨海默病等痴呆症）。如果可以使计算机表示近似于进行搜索的人的语义表示，那么感知和语言信息的集成也可以导致更好的信息搜索应用算法（例如，互联网搜索引擎）。

Humans learn about the meanings of words and larger discourse units from repeated experience with both linguistic and perceptual information. However, current computational models of semantic learning and representation focus only on linguistic structure. This CAREER award explores how humans use multisensory perception and linguistic experience to organize semantic memory. Dr. Michael Jones at Indiana University will use specially designed web-based experiment protocols to collect large amounts of data about the perceptual structure of word referents. For example, a participant presented with a target word will be required to produce a list of verbal properties to describe the word (e.g., given DOG, a participant might produce "has four legs, has fur, barks" etc.). A second participant is then provided with only the list of properties and is required to guess what word is being described. Alternatively, the information to the second participant can be a drawing of what the target word represents to the first participant, who has configured predefined object components into a 2D or 3D display. By designing these experiments as internet based, Dr. Jones will collect very large amounts of data to design and test computer models of linguistic and perceptual integration during word learning. The models resulting from this research may provide a better understanding of the learning disorders characterized by language deficiencies (e.g. many forms of autism) and age-related disorders characterized by semantic disorganization (e.g., dementias such as Alzheimer's Disease). The integration of perceptual and linguistic information could also lead to better applied algorithms for information search (e.g., Internet search engines) if the computer representation can be made to approximate the semantic representation of the human doing the searching.