US-French Collaboration: Collaborative Research: Neuro-Computational Models of Natural Language

我们的社会是建立在共享思想的基础上的，这些思想通过“被理解”的语言从一个人传播到另一个人。但是大脑是如何赋予我们理解一连串口语的能力的呢？这是计算神经科学中的一个重大挑战问题。这个项目使用语言理解过程的数学模型来解决这个问题。这些模型反映了计算机科学和语言学的见解。它们允许研究人员问：哪个过程模型最能解释特定大脑区域在特定时刻的信号？这些信号来自于人们听同一本书的法语和英语版本。通过比较不同的模型和不同的语言，该项目试图区分理解过程中特定于语言的方面和可能是全人类共同的方面。这类日益精确的建模为未来研究使用语言有困难的人铺平了道路，比如那些患有自闭症谱系障碍的人。它还可能带来更好的计算机系统，那些以大脑启发的方式使用语言的计算机系统。这个项目将计算语言学家和认知神经学家联系在一起，寻求两个具体的问题：(1)句子结构的哪些方面决定了我们对即将到来的单词的期望？(2)在自然语言中，记忆和写作之间的具体平衡是什么？使用脑电(EEG)和功能磁共振成像(FMRI)，PI检查参与者对一部文学作品的口头朗诵的神经反应。这些神经信号由时间序列预测器进行拟合，这些预测器本身来自语言上可信的语法和其他语言模型。该项目探索了一系列这样的模型，改变了语法单位的大小以及这些单位被简单记忆而不是逐步建立的倾向。通过信息理论复杂性度量，这些理论得出了对听故事的人的时刻神经反应的定量预测。作为一个整体，这种方法导致了计算上明确的过程模型，这些模型基于人类大脑对两种语言的自然文本的反应。法国国家研究机构(ANR)正在资助一个配套项目。

Our society is built upon shared ideas, ideas that get from one person to another via language that is "understood." But how do brains give us the ability to understand a stream of spoken words? This is a grand challenge question in computational neuroscience. This project addresses it using mathematical models of the language understanding process. These models reflect insights from computer science as well as linguistics. They allow investigators to ask: which process model best accounts for the signals from a particular brain region, at particular moment in time? The signals come from people listening to French and English versions of the same book.  By comparing across models and across languages, the project seeks to differentiate between aspects of the understanding process that are language-specific and aspects that might be common to all humans. Increasingly precise modeling of this sort paves the way for future work with individuals who have trouble using language, such as those with Autism Spectrum Disorder. It could also lead to better computer systems, ones that use language in a brain-inspired way.Bringing together computational linguists and cognitive neuroscientists, this project pursues two specific questions: (1) what aspects of sentence structure determine our expectations for upcoming words? and (2) what is the detailed balance between memorization and composition in natural language? Using electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI) the PIs examine participants' neural responses to the spoken recitation of a literary work. These neural signals are fitted by time series predictors, themselves derived from linguistically plausible grammars and other language models. The project explores a family of such models, varying the size of grammatical units as well as the propensity for such units to be simply memorized as opposed to built up, step by step. Via information-theoretical complexity metrics, these theories derive quantitative predictions about the moment-by-moment neural responses of a person hearing a story.  The approach as a whole leads to computationally explicit process models that are grounded in human brain responses to naturalistic text across two languages.A companion project is being funded by the French National Research Agency (ANR).