CDI-TYPE II: From Language to Neural Representations of Meaning

该项目旨在通过将脑成像、机器学习和计算建模的观点结合起来，使用行为心理学、语言学、计算机科学和神经科学的融合方法，对大脑如何表示和操纵意义进行新的理解。特别是，研究了对单词、短语和句子的含义进行编码的大脑活动，以及大脑如何根据它们的组成语义特征对单个单词的含义进行编码，当单个单词出现在短语或分句中时如何修改其编码，以及如何从其组成词的编码中构建短语或分句的编码。这项工作建立在最近的研究基础上，这些研究表明：(1)fMRI激活的可重复模式与观看描述具体物体的名词（如“锤子”或“脚趾”）有关；(2)编码这些单词含义的神经模式在不同的人之间是相似的；(3)无论人们看到的是一个单词还是物体的图片，这些编码都是相似的。鉴于之前的工作主要集中在孤立的单个单词的神经表示上，该项目研究了多个单词短语和句子，它们包含更大的知识单元；例如，名词的神经编码是如何受到形容词的影响的（例如，“fast rabbit”和“fast rabbit”）。“可爱的兔子”)，以及命题的神经编码如何与其组成词的编码相关（“cut”和“surgeons”如何在命题“surgeons cut”中组合）。为了解决这些问题，计算模型的开发使用了一组不同的训练数据，包括功能磁共振成像数据，来自代表典型语言使用的万亿字文本语料库的数据，来自语言理解和判断任务的行为数据，以及在线语言知识库（如vernet），以及来自认知神经科学文献中关于大脑如何以及在何处编码意义的理论建议。这些观点以一种计算模型的形式被整合到一个理论中，这种计算模型是从不同的数据和先验知识中训练出来的，并且能够对与成千上万个单词、成千上万个短语和句子相关的神经编码和行为反应做出实验可测试的预测。这个项目有可能在理解大脑和心智之间的关系方面取得重大的科学进展，影响语义学研究涉及的各种科学学科，包括语言学、心理学、哲学和认知科学。第二个影响来自于使用这些方法和结果来理解涉及语言障碍的大脑病理，如失语症、阅读障碍和自闭症。第三个影响来自新的统计机器学习算法的发展，用于分析和建模跨领域数据集，以帮助科学发现。最后，新出现的结果和方法将通过由首席研究员教授的脑成像、机器学习和心理学课程，以及通过专门开发的“意义的神经表征”主题的新课程产生教育影响，这些课程的材料将在网上提供。

This project seeks to develop a new understanding of how the brain represents and manipulates meaning, by bringing together the perspectives of brain imaging, machine learning and computational modeling, using converging approaches from behavioral psychology, linguistics, computer science and neuroscience. In particular, the brain activity that encodes the meanings of words, phrases and sentences is studied, along with how the brain encodes the meaning of individual words in terms of their component semantic features, how it modifies its encoding of an individual word when it occurs within a phrase or clause, and how it constructs the encoding of a phrase or clause from the encodings of its component words. This work builds on recent research showing (1) that repeatable patterns of fMRI activation are associated with viewing nouns describing concrete objects such as "hammer" or "toe," (2) that the neural patterns that encode the meanings of these words are similar across different people, and (3) that these encodings are similar whether the person views a word or a picture of the object. Whereas previous work has focused on the neural representation of single words in isolation, this project studies multiple word phrases and sentences, which comprise larger units of knowledge; for example how the neural encoding of a noun is influenced by its adjective (e.g., "fast rabbit" vs. "cuddly rabbit") and how the neural encoding of a proposition is related to the encodings of its component words (how "cut" and "surgeons" combine in the proposition "surgeons cut"). To address these questions, computational models are developed using a diverse set of training data including fMRI data, data from a trillion-word corpus of text that represents typical language use, and behavioral data from language comprehension and judgment tasks, as well as online linguistic knowledge bases such as VerbNet, and theoretical proposals from the cognitive neuroscience literature regarding how and where the brain encodes meaning. These perspectives are integrated into a theory in the form of a computational model trained from diverse data and prior knowledge, and capable of making experimentally testable predictions about the neural encodings and behavioral responses associated with tens of thousands of words, and hundreds of thousands of phrases and sentences.This project potentially constitutes a significant scientific advance in understanding the relation between brain and mind, impacting a variety of scientific disciplines involved in the study of semantics, including linguistics, psychology, philosophy and cognitive science. A second impact comes from use of the methods and results to understand brain pathologies that involve language disturbances, such as aphasia, dyslexia, and autism. A third impact comes from the development of new statistical machine learning algorithms for analyzing and modeling cross-domain data sets to aid in scientific discovery. Finally, the emerging results and methods will have an educational impact through courses on Brain Imaging, Machine Learning, and Psychology taught by the Principal Investigators, and through a new course to be developed specifically on the topic of "Neural representations of meaning," with materials to be made available on the web.