CompCog: Collaborative Research: Testing quantitative predictions of sentence processing theories with a large-scale eye-tracking database

CompCog：协作研究：使用大型眼动追踪数据库测试句子处理理论的定量预测

• 批准号: 2020914
• 负责人: Brian Dillon
• 金额: $ 25.67万
• 依托单位: University of Massachusetts Amherst
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2020
• 资助国家: 美国
• 起止时间: 2020-09-01 至 2024-08-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2020914&HistoricalAwards=false
• 关键词: CompCog; Collaborative; Research; Testing; quantitative

Modern computers are getting remarkably good at producing and understanding human language. But do they accomplish this in the same way that humans do? To address these questions, the investigators will derive measures of the difficulty of sentence comprehension by computer systems that are based on deep-learning technology, a technology that increasingly powers applications such as automatic translation and speech recognition systems. They will then use eye-tracking technology to compare the difficulty that people experience when reading sentences that are temporarily misleading, such as "the horse raced past the barn fell," with the difficulty encountered by the deep-learning systems. Based on this comparison, the researchers will modify the computer models to make them behave more like humans when processing language. This will enhance our understanding of the strategies that humans use to understand sentences while also having the potential to advance language processing technologies.The eye-tracking-while-reading measurements collected over the course of the project will be accessible to all in an open repository called the Garden Path Benchmark. This benchmark will combine the focus on syntactically challenging sentences traditionally used in psycholinguistics experiments with more recent ‘big data’ approaches to data collection and analysis. The resulting database will contain enough eye-tracking data to get clear estimates of the word-by-word processing difficulty associated with a range of constructions and specific sentences. This will allow researchers to test the quantitative predictions of deep-learning systems and other computational models at a scale that has previously not been possible. The dataset will also be used to develop parsing models that integrate contemporary deep-learning architectures with traditional symbolic parsing models from the psycholinguistics literature. This fusion will make it possible to incorporate scientific assumptions about human cognitive processes, such as reanalysis (the revision of the interpretation of a sentence when it turns out that the reader’s first interpretation was incorrect), into the neural networks. Both the Garden Path Benchmark and the models developed will be released as open access to other researchers, to support further efforts to align machine learning models and human language processing models.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

现代计算机在产生和理解人类语言方面正变得非常出色。但它们能像人类一样做到这一点吗？为了解决这些问题，调查人员将得出基于深度学习技术的计算机系统理解句子的难度的衡量标准，这种技术越来越多地支持自动翻译和语音识别系统等应用程序。然后，他们将使用眼球跟踪技术，将人们在阅读暂时具有误导性的句子时所经历的困难与深度学习系统所遇到的困难进行比较，例如“马跑过谷仓坠落”。基于这种比较，研究人员将修改计算机模型，使其在处理语言时表现得更像人类。这将加强我们对人类用来理解句子的策略的理解，同时也有可能推动语言处理技术的进步。在项目过程中收集的眼球跟踪阅读测量结果将在一个名为花园路径基准的开放储存库中向所有人开放。这一基准将把重点放在心理语言学实验中传统上具有句法挑战性的句子上，并结合最近的大数据方法来收集和分析数据。由此产生的数据库将包含足够的眼球跟踪数据，以清楚地估计与一系列结构和特定句子相关的逐字处理难度。这将使研究人员能够在以前不可能的规模上测试深度学习系统和其他计算模型的定量预测。该数据集还将用于开发分析模型，该模型将当代深度学习体系结构与心理语言学文献中的传统符号分析模型相结合。这种融合将使将关于人类认知过程的科学假设纳入神经网络成为可能，例如重新分析(当读者的第一次解释被证明是错误的时，对句子的解释进行修改)。Garden Path基准和开发的模型都将作为开放获取发布给其他研究人员，以支持进一步协调机器学习模型和人类语言处理模型的努力。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Syntactic Surprisal From Neural Models Predicts, But Underestimates, Human Processing Difficulty From Syntactic Ambiguities

神经模型的句法惊喜预测但低估了句法歧义带来的人类处理难度

• DOI: 10.18653/v1/2022.conll-1.20
• 发表时间: 2022
• 期刊: Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL
• 作者: Arehalli, Suhas;Dillon, Brian;Linzen, Tal
• 通讯作者: Linzen, Tal

Single‐Stage Prediction Models Do Not Explain the Magnitude of Syntactic Disambiguation Difficulty

单阶段预测模型无法解释句法消歧困难的严重程度

• DOI: 10.1111/cogs.12988
• 发表时间: 2021
• 期刊: Cognitive Science
• 影响因子: 2.5
• 作者: van Schijndel, Marten;Linzen, Tal
• 通讯作者: Linzen, Tal