Incremental Effects of Mismatch during Picture-Sentence Integration: Evidence from Eye-Tracking
Incremental Effects of Mismatch during Picture-Sentence Integration: Evidence from Eye-Tracking
复制标题
图句整合过程中不匹配的增量效应：来自眼动追踪的证据
DOI：
--
复制
发表时间：
2005
期刊：
影响因子：
--
通讯作者：
P. Knoeferle
中科院分区：
文献类型：
--
作者：
M. Crocker;P. Knoeferle
图片-句子整合过程中不匹配的增量效应：来自眼动追踪的证据Pia Knoeferle（knoeferle@coli.uni-sb.de）萨尔兰大学计算语言学系，66041 Saarbrüucken，德国Matthew W. Crocker（crocker@coli.uni-sb.de），萨尔兰大学计算语言学系，66041 Saarbrüucken，德国，当图片和句子匹配时，它们的整合应该比图片和句子不匹配时更快。Carpenter和Just（1975）提出的一个图画-句子整合模型预测了图画-句子整合的难易程度分别取决于图画-句子的匹配程度和不匹配程度。然而，安德伍德、杰贝特和罗伯茨（2004）的最新研究发现，在句子验证研究中，没有发现序列图片-句子呈现的匹配/不匹配差异。在一个句子理解的研究与串行图片句子呈现，我们发现没有匹配/不匹配的影响，在总句子检查时间。然而，个别句子区域的检查时间揭示了在对应的图片成分不匹配的非常句子成分处的不匹配效应，并且这是在具有句子理解而不是验证任务的研究中。根据对并发场景的观察中口语句子理解的见解，我们认为安德伍德等人的研究中不存在失配效应可能是由于凝视时间分析的粒度。一种增量式句图比较模型？我们如何将我们在场景中看到的内容与我们阅读的句子结合起来？在各种不同的理解情境中，比如我们阅读漫画书（卡罗尔、杨和盖尔廷，1992）、报纸广告（雷纳、罗泰洛、斯图尔特、凯尔和达菲，2001），或者查看科学图表（费尼、霍拉、利弗塞奇、芬德利和梅特卡夫，2003）时，都有兴趣回答这个问题。Carpenter和Just（1975）提出的“成分比较模型”（CCM）解释了图片和句子是如何整合的。他们认为，人们建立了句子和图片成分的心理表征，然后将句子和图片的相应成分相互比较。他们的句子验证模型通过将反应延迟的差异归因于句子和图片之间的一致性/不一致性（例如，Gough，1965，Just & Carpenter，1971）。在句子验证任务中，Just和Carpenter（1971）向人们展示了一张红色或黑色圆点的图片，然后是一个相关的书面句子。当句子中的颜色形容词（红色）与所描绘的点的颜色（红色）相匹配时，句子验证反应延迟比不匹配时（黑色）更短。CCM准确预测CCM得到了验证任务中离线响应延迟的强有力支持，并且主要是句子-图片验证的模型。该模型规范-至少在某种程度上-图片和书面句子的整合如何渐进地进行：通过串行比较句子和相应的图片成分的表示。反应时间适用于测试图像-图像集成步骤的复杂性。然而，为了真正检查基于图片和基于图片的心理表征的增量整合，它们比其他在线测量（例如眼动跟踪）提供的信息少。在图片整合研究中，很少有研究监测句子阅读过程中的眼动（例如，卡罗尔等人，1992;安德伍德等人，2004年）。在最近的研究中，在照片验证范式，采用眼动跟踪，由安德伍德等人（2004年）的研究结果提出了挑战的有效性CCM。他们进一步确定了重要的附加因素（例如，图片-句子呈现的顺序）影响它们的整合。安德伍德等人（2004）在两项带有图片验证任务的眼动追踪研究中，研究了真实世界照片和标题的呈现顺序的影响。他们报告了整个句子和图片的总检查时间、注视次数和注视持续时间以及反应时间。在实验1中，图片和标题一起呈现，操纵一致性（匹配/不匹配）。结果证实了已建立的匹配/错配效应：错配的响应潜伏期长于匹配条件。总检查次数和注视次数进一步证实了这一发现。在实验2中，除了匹配/不匹配操作之外，还引入了呈现顺序（图片优先，图片优先）作为条件。关键的是，与实验1相反，在实验2中，整个句子的反应时间和检查时间都没有匹配/不匹配效应。应答准确率较高（匹配率分别为83.6%和79.2
Incremental Effects of Mismatch during Picture-Sentence Integration: Evidence from Eye-tracking Pia Knoeferle (knoeferle@coli.uni-sb.de) Department of Computational Linguistics, Saarland University, 66041 Saarbr¨ ucken, Germany Matthew W. Crocker (crocker@coli.uni-sb.de) Department of Computational Linguistics Saarland University, 66041 Saarbr¨ ucken, Germany that when there is a match between a picture and a sen- tence, their integration should be faster than when a picture and a sentence do not match. Abstract A model of sentence-picture integration developed by Carpenter and Just (1975) predicts that picture- sentence integration ease/difficulty depends on picture- sentence match/mismatch respectively. Recent find- ings by Underwood, Jebbet, and Roberts (2004), how- ever, fail to find a match/mismatch difference for se- rial picture-sentence presentation in a sentence veri- fication study. In a sentence comprehension study with serial picture-sentence presentation we find no match/mismatch effect in total sentence inspection times. However, inspection times for individual sentence regions reveal a mismatch effect at the very sentence constituent for which the corresponding picture con- stituent mismatches, and this in a study with a sentence comprehension rather than verification task. Drawing on insights about spoken sentence comprehension dur- ing the inspection of concurrent scenes, we suggest that the absence of a mismatch effect in the Underwood et al. studies might be due to grain size of gaze time analyses. A Model of Incremental Sentence-Picture Comparison? Introduction How do we integrate what we see in a scene with a sentence that we read? Answering this question is of interest in various types of comprehension situations such as when we read comic books (Carroll, Young, & Guertin, 1992), newspaper advertisements (Rayner, Rotello, Stewart, Keir, & Duffy, 2001), or inspect sci- entific diagrams (Feeney, Hola, Liversedge, Findlay, & Metcalf, 2003). One account of how a picture and sentence are inte- grated is the “Constituent Comparison Model” (CCM) by Carpenter and Just (1975). They suggest that peo- ple build a mental representation of sentence and picture constituents, and that the corresponding constituents of sentence and picture are then serially compared with one another. Their model of sentence verification ac- counts for response latencies in a number of sentence- picture verification studies by attributing differences in the response latencies to congruence/incongruence be- tween sentence and picture (e.g., Gough, 1965, Just & Carpenter, 1971). In a sentence verification task, Just and Carpenter (1971) presented people with a picture of either red or black dots, followed by a related written sentence. Sen- tence verification response latencies were shorter when the colour adjective in the sentence (red ) matched the colour of the depicted dots (red) than when it did not match their colour (black). The CCM predicts precisely The CCM has received strong support from off-line re- sponse latencies in verification tasks, and is primarily a model of sentence-picture verification. The model spec- ifies - at least to some extent - how the integration of picture and a written sentence proceeds incrementally: by serially comparing the representations of sentence and corresponding picture constituents. Reaction times are approriate for testing the complex- ity of sentence-picture integration steps. However, for truly examining the incremental integration of picture- and sentence-based mental representations they are less informative than other, on-line measures such as eye- tracking. In sentence-picture integration research, few studies have monitored eye-movements during sentence reading (e.g., Carroll et al., 1992; Underwood et al., 2004). Among recent studies in the sentence-picture verification paradigm that have employed eye-tracking, findings by Underwood et al. (2004) have challenged the validity of the CCM. They have further identified im- portant additional factors (e.g., order of picture-sentence presentation) that affect their integration. In two eye-tracking studies with a sentence-picture verification task, Underwood et al. (2004) examined the effect of presentation order for real-world photographs and captions. They report total inspection time, num- ber of fixations, and durations of fixations for the en- tire sentence and picture in addition to response la- tencies. In Experiment 1, picture and caption were presented together, and congruence was manipulated (match/mismatch). Results confirmed the established match/mismatch effect: Response latencies were longer for the mismatch than for the match condition. Total in- spection times and number of fixations further confirmed this finding. In Experiment 2, order of presentation (picture-first, sentence-first) was introduced as a condition in addition to the match/mismatch manipulation. Crucially, and in contrast to Experiment 1, there was no match/mismatch effect in Experiment 2 in either response latencies or in- spection times for the entire sentence. Response accu- racy was relatively high (83.6 and 79.2 percent for match
影响因子：
2.6
作者：
Rayner, K;Rotello, CM;Duffy, SA
通讯作者：
Duffy, SA