CompCog: Collaborative Research: Testing quantitative predictions of sentence processing theories with a large-scale eye-tracking database

现代计算机在产生和理解人类语言方面变得非常出色。但它们能像人类一样做到这一点吗？为了解决这些问题，研究人员将通过基于深度学习技术的计算机系统来衡量句子理解的难度，这种技术越来越多地为自动翻译和语音识别系统等应用提供动力。然后，他们将使用眼动跟踪技术来比较人们在阅读暂时误导的句子时遇到的困难，例如“马跑过谷仓”，以及深度学习系统遇到的困难。基于这种比较，研究人员将修改计算机模型，使它们在处理语言时表现得更像人类。这将增强我们对人类用于理解句子的策略的理解，同时也有可能推进语言处理技术。在项目过程中收集的眼动追踪阅读测量数据将在一个名为Garden Path Benchmark的开放存储库中向所有人开放。这个基准测试将联合收割机结合对传统上用于心理语言学实验的句法挑战性句子的关注，以及最近的“大数据”方法来收集和分析数据。由此产生的数据库将包含足够的眼动追踪数据，以获得与一系列结构和特定句子相关的逐字处理难度的清晰估计。这将使研究人员能够以前所未有的规模测试深度学习系统和其他计算模型的定量预测。该数据集还将用于开发解析模型，将当代深度学习架构与心理语言学文献中的传统符号解析模型相结合。这种融合将使人们有可能将关于人类认知过程的科学假设，如重新分析（当读者的第一次解释不正确时，对句子的解释进行修改）纳入神经网络。Garden Path Benchmark和开发的模型都将作为开放访问向其他研究人员发布，以支持机器学习模型和人类语言处理模型的进一步努力。该奖项反映了NSF的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Modern computers are getting remarkably good at producing and understanding human language. But do they accomplish this in the same way that humans do? To address these questions, the investigators will derive measures of the difficulty of sentence comprehension by computer systems that are based on deep-learning technology, a technology that increasingly powers applications such as automatic translation and speech recognition systems. They will then use eye-tracking technology to compare the difficulty that people experience when reading sentences that are temporarily misleading, such as "the horse raced past the barn fell," with the difficulty encountered by the deep-learning systems. Based on this comparison, the researchers will modify the computer models to make them behave more like humans when processing language. This will enhance our understanding of the strategies that humans use to understand sentences while also having the potential to advance language processing technologies. The eye-tracking-while-reading measurements collected over the course of the project will be accessible to all in an open repository called the Garden Path Benchmark. This benchmark will combine the focus on syntactically challenging sentences traditionally used in psycholinguistics experiments with more recent ‘big data’ approaches to data collection and analysis. The resulting database will contain enough eye-tracking data to get clear estimates of the word-by-word processing difficulty associated with a range of constructions and specific sentences. This will allow researchers to test the quantitative predictions of deep-learning systems and other computational models at a scale that has previously not been possible. The dataset will also be used to develop parsing models that integrate contemporary deep-learning architectures with traditional symbolic parsing models from the psycholinguistics literature. This fusion will make it possible to incorporate scientific assumptions about human cognitive processes, such as reanalysis (the revision of the interpretation of a sentence when it turns out that the reader’s first interpretation was incorrect), into the neural networks. Both the Garden Path Benchmark and the models developed will be released as open access to other researchers, to support further efforts to align machine learning models and human language processing models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.