US-French Collaboration: Collaborative Research: Neuro-Computational Models of Natural Language

我们的社会是建立在共享的思想，思想从一个人到另一个通过语言是“理解”。“但是大脑是如何赋予我们理解一连串口语的能力的呢？这是计算神经科学中的一个重大挑战性问题。这个项目使用语言理解过程的数学模型来解决这个问题。这些模型反映了计算机科学和语言学的见解。研究人员可以借此提出这样一个问题：哪种处理模型最能解释在特定时刻来自特定大脑区域的信号？这些信号来自人们听同一本书的法语和英语版本。 通过跨模型和跨语言的比较，该项目试图区分理解过程中特定于语言的方面和可能对所有人都通用的方面。这种越来越精确的建模为未来对语言使用困难的人（如自闭症谱系障碍患者）的研究铺平了道路。它还可能导致更好的计算机系统，以大脑启发的方式使用语言的系统。该项目汇集了计算语言学家和认知神经科学家，追求两个具体的问题：（1）句子结构的哪些方面决定了我们对即将到来的单词的期望？（2）自然语言中记忆和写作的平衡是什么？使用脑电图（EEG）和功能性磁共振成像（fMRI），PI检查参与者对文学作品的口头背诵的神经反应。这些神经信号由时间序列预测器拟合，这些预测器本身来自语言学上合理的语法和其他语言模型。该项目探索了一系列这样的模型，改变了语法单位的大小，以及这些单位的倾向，简单地记忆，而不是建立起来，一步一步。通过信息理论的复杂性度量，这些理论得出了关于一个人听故事时每时每刻神经反应的定量预测。 这种方法作为一个整体导致了计算上明确的过程模型，这些模型基于人类大脑对两种语言的自然文本的反应。一个配套项目正在由法国国家研究机构（ANR）资助。

Our society is built upon shared ideas, ideas that get from one person to another via language that is "understood." But how do brains give us the ability to understand a stream of spoken words? This is a grand challenge question in computational neuroscience. This project addresses it using mathematical models of the language understanding process. These models reflect insights from computer science as well as linguistics. They allow investigators to ask: which process model best accounts for the signals from a particular brain region, at particular moment in time? The signals come from people listening to French and English versions of the same book. By comparing across models and across languages, the project seeks to differentiate between aspects of the understanding process that are language-specific and aspects that might be common to all humans. Increasingly precise modeling of this sort paves the way for future work with individuals who have trouble using language, such as those with Autism Spectrum Disorder. It could also lead to better computer systems, ones that use language in a brain-inspired way. Bringing together computational linguists and cognitive neuroscientists, this project pursues two specific questions: (1) what aspects of sentence structure determine our expectations for upcoming words? and (2) what is the detailed balance between memorization and composition in natural language? Using electroencephalography (EEG) and functional Magnetic Resonance Imaging (fMRI) the PIs examine participants' neural responses to the spoken recitation of a literary work. These neural signals are fitted by time series predictors, themselves derived from linguistically plausible grammars and other language models. The project explores a family of such models, varying the size of grammatical units as well as the propensity for such units to be simply memorized as opposed to built up, step by step. Via information-theoretical complexity metrics, these theories derive quantitative predictions about the moment-by-moment neural responses of a person hearing a story. The approach as a whole leads to computationally explicit process models that are grounded in human brain responses to naturalistic text across two languages. A companion project is being funded by the French National Research Agency (ANR).