CDI-TYPE II: From Language to Neural Representations of Meaning

这个项目试图通过将大脑成像、机器学习和计算建模的视角结合在一起，使用行为心理学、语言学、计算机科学和神经科学的融合方法，对大脑如何表达和操纵意义产生新的理解。特别是，研究了对词、短语和句子的意义进行编码的大脑活动，以及大脑如何根据单个单词的组成语义特征对其意义进行编码，当单个单词出现在短语或从句中时，大脑如何修改其编码，以及如何根据短语或从句的组成单词的编码构建短语或从句的编码。这项工作建立在最近的研究基础上，该研究表明：(1)功能磁共振成像的可重复激活模式与观看描述具体物体的名词有关，如“锤子”或“脚趾”；(2)编码这些单词含义的神经模式在不同的人中是相似的；(3)无论人看的是单词还是物体的图片，这些编码都是相似的。虽然以前的工作集中在单个单词的神经表示上，但这个项目研究了多个单词短语和句子，这些短语和句子组成了更大的知识单元；例如，名词的神经编码如何受到其形容词的影响(例如，“快兔”与“可爱的兔子”)，以及命题的神经编码如何与其组成部分的编码相关(“切割”和“外科医生”如何在命题“外科医生切割”中组合)。为了解决这些问题，使用一组不同的训练数据来开发计算模型，这些数据包括功能磁共振成像数据、来自代表典型语言使用的万亿字文本语料库的数据、来自语言理解和判断任务的行为数据、以及在线语言知识库(如VerbNet)，以及来自认知神经科学文献的关于大脑如何以及在哪里编码含义的理论建议。这些观点以计算模型的形式被整合到一个理论中，该模型由不同的数据和先验知识训练而成，能够对与数万个单词、数十万个短语和句子相关的神经编码和行为反应做出可实验测试的预测。该项目潜在地构成了理解大脑和大脑之间关系的重大科学进步，影响了涉及语义学研究的各种科学学科，包括语言学、心理学、哲学和认知科学。第二个影响来自于使用这些方法和结果来理解涉及语言障碍的大脑病理，如失语症、阅读障碍和自闭症。第三个影响来自开发新的统计机器学习算法，用于分析和建模跨域数据集，以帮助科学发现。最后，新出现的结果和方法将通过首席研究人员教授的脑成像、机器学习和心理学课程，以及将专门开发的一门专门关于“意义的神经表征”的新课程，产生教育影响，材料将在网上提供。

This project seeks to develop a new understanding of how the brain represents and manipulates meaning, by bringing together the perspectives of brain imaging, machine learning and computational modeling, using converging approaches from behavioral psychology, linguistics, computer science and neuroscience. In particular, the brain activity that encodes the meanings of words, phrases and sentences is studied, along with how the brain encodes the meaning of individual words in terms of their component semantic features, how it modifies its encoding of an individual word when it occurs within a phrase or clause, and how it constructs the encoding of a phrase or clause from the encodings of its component words. This work builds on recent research showing (1) that repeatable patterns of fMRI activation are associated with viewing nouns describing concrete objects such as "hammer" or "toe," (2) that the neural patterns that encode the meanings of these words are similar across different people, and (3) that these encodings are similar whether the person views a word or a picture of the object. Whereas previous work has focused on the neural representation of single words in isolation, this project studies multiple word phrases and sentences, which comprise larger units of knowledge; for example how the neural encoding of a noun is influenced by its adjective (e.g., "fast rabbit" vs. "cuddly rabbit") and how the neural encoding of a proposition is related to the encodings of its component words (how "cut" and "surgeons" combine in the proposition "surgeons cut"). To address these questions, computational models are developed using a diverse set of training data including fMRI data, data from a trillion-word corpus of text that represents typical language use, and behavioral data from language comprehension and judgment tasks, as well as online linguistic knowledge bases such as VerbNet, and theoretical proposals from the cognitive neuroscience literature regarding how and where the brain encodes meaning. These perspectives are integrated into a theory in the form of a computational model trained from diverse data and prior knowledge, and capable of making experimentally testable predictions about the neural encodings and behavioral responses associated with tens of thousands of words, and hundreds of thousands of phrases and sentences.This project potentially constitutes a significant scientific advance in understanding the relation between brain and mind, impacting a variety of scientific disciplines involved in the study of semantics, including linguistics, psychology, philosophy and cognitive science. A second impact comes from use of the methods and results to understand brain pathologies that involve language disturbances, such as aphasia, dyslexia, and autism. A third impact comes from the development of new statistical machine learning algorithms for analyzing and modeling cross-domain data sets to aid in scientific discovery. Finally, the emerging results and methods will have an educational impact through courses on Brain Imaging, Machine Learning, and Psychology taught by the Principal Investigators, and through a new course to be developed specifically on the topic of "Neural representations of meaning," with materials to be made available on the web.