CAREER: Uncovering the brain circuitry of language and its interaction with other modalities

这个项目旨在更好地了解不同的大脑区域是如何相互作用的，以便在复杂的现实生活环境中理解和产生语言。语言是人类经验的重要组成部分，用途广泛，与其他能力密切相关。例如，阅读一个故事不仅依赖于语言处理，还依赖于社会推理、对后续事件的推理、视觉意象等。然而，由于语言的复杂性，大多数神经成像实验都集中在语言的一个方面(如句法)，并将语言与其他认知能力分开研究。这导致了关于哪些大脑区域参与语言处理以及每个区域的广泛作用的理论，但还无法揭示每个区域处理哪些单独的意义单位，信息如何在区域之间流动，以及这些区域如何与处理其他认知能力的区域相互作用，如视觉或社会推理。这一建议汇集了自然环境中更强大的语言实验数据，并利用机器学习、统计学和人工智能的进展来研究现实世界的语言使用。该项目旨在进一步了解语言的神经生物学及其与其他形式的相互作用，同时通过联合建模大脑活动和刺激来改进人工智能方法。目标1将侧重于使用皮质内记录(立体脑成像)以精细的空间和时间分辨率追踪语言系统中的信息流，从而能够更准确地发现语言特征在哪里被处理以及信息如何在区域之间传递。Aim 2将语言与视觉和社交处理联系起来，使用来自观看热门情景喜剧的个人的广泛功能磁共振成像(FMRI)数据来模拟语言系统和其他大脑系统之间的互动。目标3将把这一方法扩展到将大脑活动与行为联系起来。一个新的功能磁共振数据集，参与者将参与自然对话，将被收集并用于研究潜在的高级大脑表征。拟议的多学科研究将为学生带来新的教育材料以及探索使用机器学习和人工智能促进科学的新机会。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This project aims to better understand how different brain areas interact with each other to enable understanding and production of language in complex real-life settings. Language is a crucial part of human experience, versatile and intimately connected to other abilities. For example, reading a story relies not only on language processing but also on social reasoning, inference of subsequent events, visual imagery, etc. However, due to the complexity of language, most neuroimaging experiments have focused on one aspect of language (e.g., syntax) and studied language in isolation from other cognitive abilities. This has led to theories about which brain areas are involved in language processing and the broad role of each area, but has not been able to reveal what individual units of meaning are processed in each area, how information flows between areas, and how these areas interact with other areas that process other cognitive abilities such as vision or social reasoning. This proposal brings together more powerful experimental data of language in a natural context and uses machine learning, statistics and advances in artificial intelligence to study real-world language use.This project seeks to further understanding of the neurobiology of language and its interaction with other modalities, while also improving AI methods by jointly modeling brain activity and stimulus. Aim 1 will focus on using intracortical recordings (stereoencephalography) to trace the flow of information in the language system at a fine spatial and temporal resolution, enabling more precise discovery of where language features are processed and how information passes between regions. Aim 2 will connect language to visual and social processing, using extensive functional magnetic resonance imaging (fMRI) data from individuals watching a popular sitcom to model the interactions between the language system and other brain systems. Aim 3 will extend this approach to relating brain activity to behavior. A new fMRI dataset where participants engage in natural conversation will be collected and used to study the underlying high-level brain representations. The proposed multidisciplinary research will lead to new educational material as well as new opportunities for students to explore the use of machine learning and AI for science.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.