Words and pictures: understanding how people gather information conveyed jointly through text and image in comics

文字和图像一起用来传达我们完成行为目标所需的广泛信息，包括对我们的安全(安全标志)、我们对世界的理解和互动(指导手册、地图、方向标志)、我们所做的选择(广告)和娱乐(卡通、漫画)至关重要的信息。在这些情况下，不仅文字和图像结合在一起，而且通常只有通过理解文字和图像的组合-它们共同传达的信息-我们才能理解正在向我们传达的是什么。尽管文字-图像组合在我们周围的世界中无处不在，我们依赖于理解它们所传达的信息，但人们对我们如何看待和理解这类刺激知之甚少。我们的大部分知识仅限于我们如何看待和理解文字或图像的文字和段落。从这些以前的方法中，我们已经形成了详细的理解，在许多方面，我们检查图像和文字的方式是非常不同的：有不同的因素影响我们看什么，甚至可能是我们阅读时与观看图像时不同的潜在加工风格和认知过程。这些关于观看和处理文字和图像的截然不同的描述引发了关于我们如何收集和理解同时包含文字和图像的刺激的问题。拟议的工作旨在为这一目前不发达的研究领域提供关键的见解。为了更好地理解我们如何检查和理解结合了文字和图像的刺激，必须解决三个关键问题：(1)我们使用什么策略来从文字和图像组合中收集信息，这些策略是否会随着单词和图像之间的关系而变化？也就是说，当词语相对于图像所传达的信息量发生变化时，我们是否会改变对这些刺激的看法？(2)我们如何理解和回应文字-图像组合中的空间？单词之间的空格传达的信息与图像中对象之间的空格非常不同。(3)在直接观察文字-图像组合之前，我们需要处理多少信息？我们在外围视觉中处理单词和图像的某些方面，但我们处理的程度存在争议。漫画提供了解决这三个问题的理想媒介，因为它们使用文字和图像的组合，在几个面板和页面上向读者传达一个故事，这些故事必须被一起理解，才能让读者理解所传达的信息。重要的是，漫画中的文字和图像的结合将故事传达给读者，在典型的漫画中，文字和图像都不能单独传达整个故事，而且存在一系列在文字和图像之间传播信息的方式。漫画不仅为探索这些研究问题提供了一个理想的媒介，而且通过这样做，我们能够使用我们的发现来为漫画理论提供新的见解。与理解我们如何阅读和理解漫画相关的研究领域越来越多，理论正在涌现，这些理论将创作漫画时图像和文字的使用方式与读者如何看待和理解漫画的体验联系起来。此外，这些想法往往借鉴心理学现象和理论，但还没有在心理学研究中或用实验心理学的科学方法来探索。目前，漫画理论缺乏实证检验。我们的工作将是对支撑漫画理论的假设的第一次测试，因此将为该领域提供至关重要的新见解，并有助于塑造这一新兴领域未来的发展方向，特别是将为该领域创造一个心理上严格的研究领域。

Word and image are used together to convey a wide range of information that we need to complete our behavioural goals, including information vital to our safety (safety signage), our understanding and interaction with the world (instruction manuals, maps, directional signage), the choices we make (advertising) and entertainment (cartoons, comics). Not only are words and images combined in these situations, but it is often only through understanding the combination of word and image - the information that they jointly convey - that we understand what is being communicated to us. Despite the pervasive nature of word-image combinations in the world around us and our reliance on understanding the information that they convey, little is understood about how we view and understand this class of stimulus. Most of our knowledge is limited to how we view and understand words and passages of text or images alone. From these previous approaches, detailed understanding has been developed, and in many ways the manner in which we inspect images and words is very different: with different factors influencing what we look at, and perhaps even different underlying processing styles and cognitive processes when we are reading compared to when we are viewing images. These very different accounts of viewing and processing words and images raise questions about how we gather and understand stimuli that contain both word and image. The proposed work aims to provide key insights into this currently under-developed field of research. Three key questions must be addressed to better understand how we inspect and understand stimuli that combine word and image: (1) What strategies do we use to gather information from word-image combinations and do these change depending on the relationship between the words and image? That is, do we vary how we view these stimuli when the relative amount of information conveyed by the words compared to the image changes? (2) How do we understand and respond to space in word-image combinations? Space between words conveys very different information than space between objects in an image. (3) How much information do we process before looking directly at a word-image combination? We process some aspects of words and images in peripheral vision, but the level to which we do so is controversial. Comics offer an ideal medium for addressing these three questions because they use word-image combinations to convey a story to the reader, over several panels and pages, which must be understood together for the reader to understand the conveyed information. Critically, it is the combination of word and image in comics that conveys the story to the reader, with neither word nor image alone conveying the entire story in a typical comic, and there exist a range of ways in which the information is distributed between words and image. Not only do comics offer an ideal medium for exploring these research questions, but in doing so we are able to use our findings to provide new insights into comics theory. There is a growing research field associated with understanding how we read and understand comics and theories are emerging that relate the manner in which images and words are used when creating comics to the reader's experience in terms of how they view and understand comics. Moreover, these ideas often drawn upon psychological phenomena and theory, yet have not been explored in within Psychological research or with the scientific method that underlies Experimental Psychology. At present comics theories lack empirical testing. Our work will be a first test of the assumptions that underpin comics theory and will therefore offer crucial new insights for this field and help shape the direction that this emerging field takes in the future and in particular will create a psychologically rigorous domain of research for this field.