Influence of spoken text on cognitive processing of complex pictorial presentations

基于多媒体学习模型(Gaser&Schwan，2015；Mayer，2014)和关于图片和场景知觉的心理学发现(Huettig&Altmann，2005)，计划中的项目考察了伴随音频文本对复杂图片内容的知觉和加工以及相关知识获取过程的影响。在四个实验室实验中，将分析音频文本在何种程度上(1)系统地降低了图片的解释开放性，(2)促进了对被忽视的图片元素的感知和加工，(3)在知觉和认知上将图片元素联系在一起，以及(4)促进了图片内容的双重编码。研究对象是关于历史事件的复杂而详细的历史绘画，通常用于传达历史知识(Popp，2013)。在这四项研究中，将考察听觉引导性文本(称为显著图像元素)、图像元素之间的关系以及文本结构对注意分配(凝视连贯性、注视次数、转换量，通过眼动测量)和知识获取(自由视觉回忆、文本-图像关系、迁移、记忆相关个体间一致性)的影响。

Based on models of multimedia learning (Glaser & Schwan, 2015; Mayer, 2014) and psychological findings on picture and scene perception (Huettig & Altmann, 2005), the planned project examines the influence of accompanying audio texts on the perception and processing of complex pictorial contents and related knowledge acquisition processes. In four laboratory experiments it will be analyzed to which extent audio texts (1) systematically reduce the interpretational openness of pictures, (2) foster the perception and processing of neglected pictorial elements, (3) perceptually and cognitively relate pictorial elements, and (4) contribute to the dual coding of the pictorial contents. Research objects are complex and detailed history paintings about historical events, which are typically used to communicate historical knowledge (Popp, 2013). In the four studies, the influence of auditive intro texts, named salient pictorial elements, named relations between pictorial elements as well as the influence of text structure on the allocation of attention (gaze coherence, fixation times, amount of transitions, measured by eye tracking) and knowledge acquisition (free visual recall, text-picture-relation, transfer, memory related interindividual consistency) will be examined.