CRCNS US-Spain Research Proposal: Collaborative Research: Tracking and modeling the neurobiology of multilingual speech recognition

世界上一半以上的人口能流利地说两种或两种以上的语言。会一门以上的语言可以让你通过对话和阅读与来自其他国家和文化的人进行交流和互动。它也有经济效益，甚至可能对认知和健康有好处。从科学上讲，双语在教育（我们如何才能最好地训练人们使用新语言？）、健康（你如何才能最好地治疗脑损伤后有语言障碍的双语者？）和技术（我们如何使语音识别系统像多语言人类一样灵活？）方面有许多悬而未决的问题。为了更好地理解多语言语言处理，研究人员将记录懂一种、两种或三种语言的人在听他们懂的语言时对语言的神经反应。该项目是美国研究人员与西班牙研究人员合作的跨大西洋合作项目。高中、本科、研究生和博士后学员将接受尖端计算和认知神经科学和心理语言学的培训，包括可转移到一系列学术和非学术STEM职业的数据科学和建模技能。研究人员将探索的一个基本问题是，两种或两种以上的语言如何在单一语言系统中共存，以及每种语言如何在大脑中表现出来。一些先前的研究表明，即使是在单一的语言环境中，一个人所知道的所有语言都有一个深度持续的协同激活，而其他研究表明，在许多情况下，只有当时相关的语言被激活。该项目将使用计算神经科学的工具来发展认知理论和实现双语和三语语言处理的模型，研究小组将与高时间分辨率的神经成像数据（脑磁图或MEG）进行比较。当单语、双语和三语个体在特定条件下处理他们所知道的语言语音时，脑磁图将被收集，这些条件被设计成促进对单一语言（来自一种语言的孤立单词或连续语音）或两种语言（来自两种语言的孤立单词随机交错，或两种语言之间更生态的“代码转换”）的注意。研究人员将使用在以前的NSF支持下开发的最先进的人类语音识别神经网络模型。他们将使用连续的语音跟踪来将神经活动与关于语言共同激活的潜在影响的理论假设以及神经网络模型的行为和内部活动联系起来。通过这种方式，研究人员将能够比较人类大脑的反应和神经网络模型对连续语音呈现过程中每个连续语音（辅音或元音）和单词的期望水平的统计预测的反应。将不同的模型与神经反应进行比较，将有助于研究人员解决一些基本问题，比如，一个人知道的所有语言是否在听到任何语言时都是活跃的，以及这是有益的还是会造成干扰。这项研究有望加深我们对人类大脑中多语言发展和处理的理解。西班牙国家研究机构（AEI）正在资助一个伙伴项目。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

More than half of the world's population speaks two or more languages fluently. Speaking more than one language allows you to communicate and interact with individuals in other countries and cultures through conversation and reading. It also has economic benefits, and possibly even cognitive and health benefits. Scientifically, there are many open questions about bilingualism for education (how can we best train people in new languages?), health (how can you best treat a bilingual person with language difficulty after a brain injury?), and technology (how can we make speech recognition systems as flexible as multilingual humans?). To better understand multilingual language processing, researchers will record neural responses to speech in people who know either one, two, or three languages, while they listen to the languages they know. The project is a transatlantic collaborative effort with US researchers partnering with Spanish researchers. High school, undergraduate, graduate, and postdoctoral trainees will receive training in cutting-edge computational and cognitive neuroscience and psycholinguistics, including data science and modeling skills transferable to a range of academic and non-academic STEM careers.One of the fundamental questions that researchers will explore is how two or more languages co-exist within a single language system and how each is represented in the brain. Some prior research suggests there is a deep continuous coactivation of all the languages a person knows even when they are in a single language context, while other research suggests that under many circumstances, only the language relevant in the moment is activated. The project will use the tools of computational neuroscience to develop cognitive theories and implemented models of bilingual and trilingual language processing, which the research team will compare to neuroimaging data with high temporal resolution (magnetoencephalography or MEG). MEG will be collected while monolingual, bilingual, and trilingual individuals process speech from languages they know under conditions designed to promote attention to a single language (isolated words or continuous speech from only 1 language) or two languages (random interleaving of isolated words from 2 languages, or more ecological 'code-switching' between 2 languages). Researchers will use a state-of-the-art neural network model of human speech recognition developed with previous NSF support. They will use continuous speech tracking to relate neural activity to both theoretically-generated hypotheses regarding potential impacts of language co-activation and the behavior and internal activity of neural network models. In this way, researchers will be able to compare human brain responses and neural network model responses to statistical predictions of the expectation level for each successive speech sound (consonant or vowel) and word during presentation of continuous speech. Comparing different models to neural responses will help researchers address fundamental questions, such as whether all the languages a person knows are active whenever they hear any language, and whether this is helpful or causes interference. This research promises to deepen our understanding of multilingual language development and processing in the human brain.A companion project is being funded by the State Research Agency, Spain (AEI).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.