Collaborative Research: NCS-FO: Studying language in the brain in the modern machine learning era

该项目将研究大脑如何处理语言，这是我们可以提出的最重要的问题之一。语言技能对终生收入和社会差距有很大影响。语言的丧失或发展的停滞可能是毁灭性的。与此同时，来自大脑的洞察可以提高机器对语言的理解，从而开辟了新的应用--从网络搜索到语音助手，再到有朝一日可以在我们的日常生活中帮助我们的机器人。使用神经科学来理解语言使用过程中发生的事情，什么是对的，什么是错的，什么是大脑使用的语言结构和理论，这将是革命性的。为了做到这一点，神经科学家使用了许多与机器学习工具相同的工具。使用大型数据集，这些机器学习工具得到了极大的改进，改变了机器的能力；然而，语言的神经科学在很大程度上无法收获这些回报。我们将提供这些数据、新的方法和指标，以使神经科学能够扩大规模并利用现代机器学习。与此同时，机器学习的规模使工具的使用民主化；科学界可以调查与他们相关的问题。今天，只有少数几个组织拥有收集数据和调查语言神经科学问题的资源，让许多社区蒙在鼓里。一个大规模的数据、工具和基准的中央储存库将使对大脑中语言的研究大众化，这是使我们成为人类的核心方面之一。我们的技术目标是产生最大的数据集，是研究语言神经科学的1000倍，以及利用这些数据的新型模型，以及将语言问题形式化以获得对语言网络和语言结构的洞察的基准。到目前为止，语言神经科学的研究只能在不同的数据集上提供语言网络的小快照，这使得构建一幅连贯的图景变得困难。一个具有精确基准的大型单一数据集正式定义了语言学中关于神经数据的假设意味着什么，这将使社区能够对相同的数据提出许多问题，从而能够综合语言网络的结构和运作。同时，人们知道需要大规模的数据来探索人工语言模型的理解。很可能，如果需要数万个句子来探索人工语言神经网络并得出有意义的见解，那么每个受试者将需要相同规模的数据来探索生物语言神经网络。围绕公共数据集基准的形式化问题已经在许多领域取得了天文数字的进展，从解析(Penn Treebank)到图像识别(ImageNet)；我们将把同样的方法应用于语言的神经科学。这个过程如此高效，部分原因是它以非领域专家可以访问的方式提出问题；机器学习专家不需要关心语言细节，他们将能够改进对大脑中语言的解码，从而通过遵循现有协议来推动洞察力。通过以精确定义的方式提出语言问题，我们还将实现跨学科合作：语言学家将能够提出基准，这些基准是围绕分类器性能或网络与神经活动之间的映射提出的问题。这些基准将提供一种通用的数学语言，不同的领域可以用这种语言来表达它们的关键问题，这种方式以前是不可能的，因为甚至不存在能够支持这类工作的数据集。我们看到的未来是，神经科学、语言学、自然语言处理和机器学习作为一个综合整体，提出关于大脑中语言的正确问题，开发支持回答这些问题的新工具，并探索支持构建连贯的语言系统图景的大规模资源。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The project will investigate how the brain processes language, one of the most consequential questions we can ask. Language skills significantly affect lifetime income and social disparities. The loss of language or a halt in its development can be devastating. At the same time, insights from the brain that could improve machines’ understanding of language opens up new applications -- from web search, to voice assistants, to, one day, robots that can help us in our daily lives. Using neuroscience to understand what happens during language use, what goes right and wrong, what linguistic structures and theories are used by the brain, would be revolutionary. To do this, neuroscientists use many of the same tools as those created for machine learning. Those machine learning tools have improved tremendously using large datasets, changing what machines are capable of; yet the neuroscience of language has been largely unable to reap these rewards. We will provide that data, the new methods and metrics, required to enable neuroscience to scale up and take advantage of modern machine learning. At the same time, scale in machine learning has democratized access to tools; scientific communities can investigate questions that pertain to them. Today, only a few groups have the resources to collect data and investigate questions around the neuroscience of language, leaving many communities in the dark. A large-scale central repository of data, tools, and benchmarks will democratize access to the study of language in the brain, one of the core aspects of what makes us human.Our technical goal is to produce the largest dataset, by a factor of 1000, for investigating the neuroscience of language along with new types of models that exploit this data, and benchmarks which formalize linguistic questions to derive insights about the language network and the structure of language. Thus far, investigations in the neuroscience of language have only been able to provide small snapshots of the language network on different datasets, making it hard to build a coherent picture. A single large-scale dataset with precise benchmarks that formally define what hypotheses in linguistics mean in terms of neural data will enable the community to ask many questions of the same data, allowing for a synthesis of the structure and operation of the language network. At the same time, large-scale data is known to be required to probe the understanding of artificial language models. It is likely that if tens of thousands of sentences are required to probe an artificial language-neural-networks and derive meaningful insight, the same scale of data will be required per subject to probe biological language-neural-networks. Formalizing questions around benchmarks on a common dataset has resulted in astronomical progress in many fields from parsing (Penn Treebank) to image recognition (ImageNet); we will apply this same methodology to the neuroscience of language. This process is so efficient, in part, because it casts questions in a way that non-domain experts can access; machine learning experts need not concern themselves with linguistic minutia, they will be able to improve decoding of language from the brain and thereby drive insights by following existing protocols. By putting forward linguistic questions in a precisely-defined manner, we will also enable cross-disciplinary collaboration: linguists will be able to propose benchmarks which are questions around the performance of classifiers or mapping between networks and neural activity. These benchmarks will provide a common mathematical language by which different fields can express their key questions, in a way that has not been possible before because no dataset existed that could even support such work. We see a future where neuroscience, linguistics, natural language processing, and machine learning act as an integrated whole to ask the right questions about language in the brain, to develop new tools that support answering those questions, and to probe a large-scale resource that supports building a coherent picture of the language system.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.