US-French Collaboration: Collaborative Research: Neuro-Computational Models of Natural Language

我们的社会建立在共同的思想之上，这些思想通过“可理解”的语言从一个人传递到另一个人。但是，大脑是如何赋予我们理解一连串口语的能力的呢？这是计算神经科学中的一个重大挑战。这个项目使用语言理解过程的数学模型来解决这个问题。这些模型反映了计算机科学和语言学的见解。它们让研究人员可以问：哪个过程模型最能解释特定大脑区域在特定时刻发出的信号？这些信号来自于人们听同一本书的法语和英语版本。通过跨模型和跨语言的比较，该项目试图区分理解过程中特定于语言的方面和所有人类可能共有的方面。这种越来越精确的模型为未来研究有语言障碍的人铺平了道路，比如那些患有自闭症谱系障碍的人。它还可能导致更好的计算机系统，以一种受大脑启发的方式使用语言。该项目汇集了计算语言学家和认知神经科学家，研究了两个具体问题：(1)句子结构的哪些方面决定了我们对即将到来的单词的期望？(2)自然语言中记忆与写作的具体平衡是什么？使用脑电图（EEG）和功能性磁共振成像（fMRI）， pi检查参与者在口头背诵文学作品时的神经反应。这些神经信号由时间序列预测器拟合，时间序列预测器本身来源于语言学上合理的语法和其他语言模型。该项目探索了一系列这样的模型，改变了语法单位的大小，以及这些单位被简单记忆的倾向，而不是一步一步地建立起来。通过信息论的复杂性度量，这些理论对一个人听到一个故事时每时每刻的神经反应进行了定量预测。该方法作为一个整体导致了基于人类大脑对两种语言的自然文本的反应的计算显式过程模型。法国国家研究机构（ANR）正在资助一个伙伴项目。

Our society is built upon shared ideas, ideas that get from one person to another via language that is "understood." But how do brains give us the ability to understand a stream of spoken words? This is a grand challenge question in computational neuroscience. This project addresses it using mathematical models of the language understanding process. These models reflect insights from computer science as well as linguistics. They allow investigators to ask: which process model best accounts for the signals from a particular brain region, at particular moment in time? The signals come from people listening to French and English versions of the same book.  By comparing across models and across languages, the project seeks to differentiate between aspects of the understanding process that are language-specific and aspects that might be common to all humans. Increasingly precise modeling of this sort paves the way for future work with individuals who have trouble using language, such as those with Autism Spectrum Disorder. It could also lead to better computer systems, ones that use language in a brain-inspired way.Bringing together computational linguists and cognitive neuroscientists, this project pursues two specific questions: (1) what aspects of sentence structure determine our expectations for upcoming words? and (2) what is the detailed balance between memorization and composition in natural language? Using electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI) the PIs examine participants' neural responses to the spoken recitation of a literary work. These neural signals are fitted by time series predictors, themselves derived from linguistically plausible grammars and other language models. The project explores a family of such models, varying the size of grammatical units as well as the propensity for such units to be simply memorized as opposed to built up, step by step. Via information-theoretical complexity metrics, these theories derive quantitative predictions about the moment-by-moment neural responses of a person hearing a story.  The approach as a whole leads to computationally explicit process models that are grounded in human brain responses to naturalistic text across two languages.A companion project is being funded by the French National Research Agency (ANR).