Developing a cross-methodlogical processing account of situated language comprehension

研究表明，不仅语言和世界知识会影响增量理解，场景信息也会影响增量理解。具体地说，三种实验范式有助于深入了解理解和视觉过程是如何相互作用的：说话理解过程中场景中的眼球跟踪，句子-图片验证任务中反应潜伏期的测量，以及情景语境中的电生理记录。然而，心理语言学的句子理解理论虽然对语言知识的整合提供了详细的解释，但还没有充分解决理解与视觉过程之间的相互作用。之前自己的工作基于口语理解过程中场景中眼睛跟踪的结果，初步描述了语言和视觉过程之间的协调作用。本研究将通过系统地利用这三种范式提供的结果和测量，来更全面地解释情景情景句子理解。通过这种方式，关于视觉注意力、处理复杂性和潜在神经过程的证据可以同时用于完善我们的帐户。不同的方法将进一步支持将帐户扩展到书面和口头语言模式，以及不同的视觉环境(简单与复杂、静态与动态)。

It has been shown that not only linguistic and world knowledge, but also scene information can influence incremental comprehension. Specifically, three experimental paradigms have contributedinsights into how comprehension and visual processes interact: eye tracking in scenes during spokencomprehension, the measuring of response latencies in sentence-picture verification tasks, and electrophysiological recordings in situated contexts. Psycholinguistic theories of sentence comprehension, however - while providing detailed accounts of the integration of linguistic knowledge- do not yet adequately address the interaction of comprehension with visual processes.Prior own work has sketched a preliminary account of the coordinated interplay between linguisticand visual processes based on findings from eye tracking in scenes during spoken comprehension.The proposed research will develop a more complete account of situated scene sentencecomprehension through the systematic exploitation of findings and measures provided by all three paradigms. In this way, evidence concerning visual attention, processing complexity, and underlying neural processes can be drawn upon simultaneously in refining our account. The diverse methods will further support extending the account across written and spoken language modalities, as well as for different visual contexts (simple vs. complex, static vs. dynamic).