CRCNS US-Spain Research Proposal: Collaborative Research: Tracking and modeling the neurobiology of multilingual speech recognition

世界上超过一半的人口能流利地说两种或两种以上的语言。会说一种以上的语言使你能够通过对话和阅读与其他国家和文化中的个人进行交流和互动。它也有经济利益，甚至可能对认知和健康有益。从科学上讲，在教育(我们如何才能最好地培训人们学习新语言？)、健康(你如何最好地治疗脑损伤后有语言障碍的双语者？)和技术(我们如何才能让语音识别系统像多语种人类一样灵活？)方面，都有许多悬而未决的问题。为了更好地理解多语言处理，研究人员将记录懂一种、两种或三种语言的人在听他们知道的语言时对语音的神经反应。该项目是美国研究人员与西班牙研究人员合作的跨大西洋合作成果。高中、本科生、研究生和博士后实习生将接受尖端计算和认知神经科学和心理语言学方面的培训，包括可转移到各种学术和非学术STEM职业的数据科学和建模技能。研究人员将探索的基本问题之一是两种或两种以上语言如何在单一语言系统中共存，以及每种语言在大脑中如何表示。先前的一些研究表明，一个人知道的所有语言即使在单一语言环境中也存在深度持续的共同激活，而其他研究表明，在许多情况下，只有与之相关的语言才会被激活。该项目将使用计算神经科学的工具来开发认知理论，并实施双语和三语语言处理的模型，研究团队将把这些模型与高时间分辨率的神经成像数据(脑磁图或脑磁图)进行比较。脑磁图将在单语、双语和三语个体处理他们所知道的语言的语音时收集，这些语言的条件旨在提高人们对单一语言(孤立的单词或仅来自一种语言的连续语音)或两种语言(来自两种语言的孤立单词的随机交错，或更生态的两种语言之间的代码切换)的注意力。研究人员将使用一种最先进的人类语音识别神经网络模型，该模型是在之前NSF的支持下开发的。他们将使用连续语音跟踪将神经活动与理论上产生的关于语言共同激活的潜在影响的假设以及神经网络模型的行为和内部活动联系起来。通过这种方式，研究人员将能够比较人脑反应和神经网络模型对连续语音呈现期间每个连续语音(辅音或元音)和单词的预期水平的统计预测。将不同的模型与神经反应进行比较，将有助于研究人员解决基本问题，例如，当一个人听到任何语言时，是否所有人知道的语言都是活跃的，这是有帮助的还是会造成干扰。这项研究承诺加深我们对人类大脑中多语言发展和处理的理解。西班牙国家研究机构(AEI)正在资助一个配套项目。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

More than half of the world's population speaks two or more languages fluently. Speaking more than one language allows you to communicate and interact with individuals in other countries and cultures through conversation and reading. It also has economic benefits, and possibly even cognitive and health benefits. Scientifically, there are many open questions about bilingualism for education (how can we best train people in new languages?), health (how can you best treat a bilingual person with language difficulty after a brain injury?), and technology (how can we make speech recognition systems as flexible as multilingual humans?). To better understand multilingual language processing, researchers will record neural responses to speech in people who know either one, two, or three languages, while they listen to the languages they know. The project is a transatlantic collaborative effort with US researchers partnering with Spanish researchers. High school, undergraduate, graduate, and postdoctoral trainees will receive training in cutting-edge computational and cognitive neuroscience and psycholinguistics, including data science and modeling skills transferable to a range of academic and non-academic STEM careers.One of the fundamental questions that researchers will explore is how two or more languages co-exist within a single language system and how each is represented in the brain. Some prior research suggests there is a deep continuous coactivation of all the languages a person knows even when they are in a single language context, while other research suggests that under many circumstances, only the language relevant in the moment is activated. The project will use the tools of computational neuroscience to develop cognitive theories and implemented models of bilingual and trilingual language processing, which the research team will compare to neuroimaging data with high temporal resolution (magnetoencephalography or MEG). MEG will be collected while monolingual, bilingual, and trilingual individuals process speech from languages they know under conditions designed to promote attention to a single language (isolated words or continuous speech from only 1 language) or two languages (random interleaving of isolated words from 2 languages, or more ecological 'code-switching' between 2 languages). Researchers will use a state-of-the-art neural network model of human speech recognition developed with previous NSF support. They will use continuous speech tracking to relate neural activity to both theoretically-generated hypotheses regarding potential impacts of language co-activation and the behavior and internal activity of neural network models. In this way, researchers will be able to compare human brain responses and neural network model responses to statistical predictions of the expectation level for each successive speech sound (consonant or vowel) and word during presentation of continuous speech. Comparing different models to neural responses will help researchers address fundamental questions, such as whether all the languages a person knows are active whenever they hear any language, and whether this is helpful or causes interference. This research promises to deepen our understanding of multilingual language development and processing in the human brain.A companion project is being funded by the State Research Agency, Spain (AEI).This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.