Words and pictures: understanding how people gather information conveyed jointly through text and image in comics

点击翻译按钮获取中文摘要

Word and image are used together to convey a wide range of information that we need to complete our behavioural goals, including information vital to our safety (safety signage), our understanding and interaction with the world (instruction manuals, maps, directional signage), the choices we make (advertising) and entertainment (cartoons, comics). Not only are words and images combined in these situations, but it is often only through understanding the combination of word and image - the information that they jointly convey - that we understand what is being communicated to us. Despite the pervasive nature of word-image combinations in the world around us and our reliance on understanding the information that they convey, little is understood about how we view and understand this class of stimulus. Most of our knowledge is limited to how we view and understand words and passages of text or images alone. From these previous approaches, detailed understanding has been developed, and in many ways the manner in which we inspect images and words is very different: with different factors influencing what we look at, and perhaps even different underlying processing styles and cognitive processes when we are reading compared to when we are viewing images. These very different accounts of viewing and processing words and images raise questions about how we gather and understand stimuli that contain both word and image. The proposed work aims to provide key insights into this currently under-developed field of research. Three key questions must be addressed to better understand how we inspect and understand stimuli that combine word and image: (1) What strategies do we use to gather information from word-image combinations and do these change depending on the relationship between the words and image? That is, do we vary how we view these stimuli when the relative amount of information conveyed by the words compared to the image changes? (2) How do we understand and respond to space in word-image combinations? Space between words conveys very different information than space between objects in an image. (3) How much information do we process before looking directly at a word-image combination? We process some aspects of words and images in peripheral vision, but the level to which we do so is controversial. Comics offer an ideal medium for addressing these three questions because they use word-image combinations to convey a story to the reader, over several panels and pages, which must be understood together for the reader to understand the conveyed information. Critically, it is the combination of word and image in comics that conveys the story to the reader, with neither word nor image alone conveying the entire story in a typical comic, and there exist a range of ways in which the information is distributed between words and image. Not only do comics offer an ideal medium for exploring these research questions, but in doing so we are able to use our findings to provide new insights into comics theory. There is a growing research field associated with understanding how we read and understand comics and theories are emerging that relate the manner in which images and words are used when creating comics to the reader's experience in terms of how they view and understand comics. Moreover, these ideas often drawn upon psychological phenomena and theory, yet have not been explored in within Psychological research or with the scientific method that underlies Experimental Psychology. At present comics theories lack empirical testing. Our work will be a first test of the assumptions that underpin comics theory and will therefore offer crucial new insights for this field and help shape the direction that this emerging field takes in the future and in particular will create a psychologically rigorous domain of research for this field.