Using Machine Learning and Cognitive Modeling to Understand the fMRI-measured Brain Activation Underlying the Representations of Words and Sentences

使用机器学习和认知建模来理解fMRI测量的单词和句子表征背后的大脑激活最近的一些fMRI研究报告了当人类受试者从不同的语义类别（例如，描述工具，建筑物或人的图片或单词）中感知描述物体的图片或单词时，fMRI大脑激活的显著和可重复的差异。目前，根据一个人的大脑活动，可以很准确地确定他正在思考的是几种语义类别中的哪一种。我们建议在这些最新发现的基础上进行新的研究，并试图理解(1)与不同语义类别的对象和动作（名词和动词）相关的人类大脑活动；(2)与语义类别相关的大脑活动是否可以划分为更原始的语义成分（例如，与工具相关的词汇相关的大脑活动是否会分解为表征工具视觉外观的一个成分和表征使用工具时涉及的运动动作的第二个成分？）；(3)在阅读单词对或简单的短语和句子时，与单个单词相关的大脑活动是如何组合成更复杂的模式的。这项研究包括：(1)应用机器学习算法来发现与特定语义域相关的全皮层大脑激活模式，(2)开发人类语言处理的计算模型，实例化所发现的表征原则，并做出具体的、可测试的预测，以及(3)进行新的功能磁共振成像研究，以获得关于人类语义类别表征的新数据。拟议研究的智力价值是多方面的。如果成功的话，我们的研究将对大脑如何组织有关词语、物体和动作的含义的信息带来新的科学见解。它还将导致fMRI数据分析的新方法，特别是发现fMRI激活的复杂时空模式，从而准确区分不同的心理状态。这项研究还将引领一种新的范式，用于开发计算认知模型，并将它们与从功能磁共振成像和行为测量中获得的经验数据相匹配。拟议研究的更广泛影响将通过向若干社区开展具体的外联活动加以扩大。除了在认知和计算神经科学文献中发表我们的科学成果外，我们还将积极参与这个社区，通过nsf资助的fMRI数据中心传播我们新的实验fMRI数据，并通过在互联网上记录和发布我们的新数据分析算法。我们将积极参与统计机器学习社区，这对开发新的功能磁共振成像分析方法有很大贡献，并将为本科和研究生教育界开发和传播教材，包括功能磁共振成像数据集。最后，我们提出的研究对医学研究界有潜在的影响，特别是在阿尔茨海默病、阅读障碍和高功能自闭症等神经系统疾病的研究方面，我们已经在这三个涉及语言障碍的领域开展了积极的研究合作，为转移可能从这项研究中产生的新科学见解提供了直接的渠道。

Using machine learning and cognitive modeling to understand the fMRI-measuredbrain activation underlying the representations of words and sentencesTom M. Mitchell and Marcel A. Just Project AbstractA number of recent fMRI studies have reported significant and repeatable differences in fMRI brain activation when human subjects perceive pictures or words describing objects from different semantic categories (e.g., pictures or words that describe tools, buildings, or people). It is currently possible to determine with good accuracy which of several semantic categories a person is thinking about, based on their brain activation.We propose new research that builds on these recent discoveries, and seeks to understand (1) human brain activity associated with different semantic categories of objects and actions (nouns and verbs); (2) whether the brain activity associated with semantic categories can be partitioned into more primitive semantic components (e.g., does the brain activity associated with words about tools factor into one component characterizing the tool's visual appearance and a second component characterizing the motor actions involved in using the tool?); and (3) how brain activity associated with individual words is combined into more complex patterns when reading word pairs or simple phrases and sentences.This research involves:(1) applying machine learning algorithms to discover cortex-wide brain activation patterns associated with particular semantic domains, (2) developing a computational model of human language processing that instantiates the representational principles discovered and that makes specific, testable predictions, and (3) conducting new fMRI studies to obtain novel data about human semantic category representations.The intellectual merit of the proposed research is multifaceted. If successful, our research will lead to new scientific insights into how the brain organizes information about meanings of words, objects, and actions. It will also lead to new methods for fMRI data analysis, especially for discovering complex temporal-spatial patterns of fMRI activation that accurately distinguish different mental states. The research will also lead toward a new paradigm for developing computational cognitive models and fitting them to empirical data obtained from fMRI and from behavioral measures.The broader impacts of the proposed research will be amplified by specific outreach activities to several communities. In addition to publishing our scientific results in the cognitive and computational neuroscience literature, we will also actively engage this community by disseminating our new experimental fMRI data through the NSF-funded fMRI Data Center, and by documenting and publishing our new data analysis algorithms on the internet. We will proactively engage the statistical machine learning community, which has much to contribute to development of new fMRI analysis methods, and will develop and disseminate teaching materials for the undergraduate and graduate educational community,including fMRI data sets. Finally, our proposed research has potential impact on the medical research community, especially regarding the study of neurological conditions such as Alzheimer's disease, dyslexia and high-functioning autism - three areas entailing a language disturbance in which we already have active research collaborations, providing a direct conduit for transferring new scientific insights that may arise from this research.