RI: Small: Modeling Vividness and Symbolism for Decoding Visual Rhetoric

点击翻译按钮获取中文摘要

This project develops systems for analyzing and inferring the non-literal messages conveyed in the media through persuasive images and text. Computational representations of two persuasive strategies are devised to model the mapping between observable information and underlying messages. First, this project models "vividness": through analyses of how human subjects perceive images and text, the system aims to identify relevant regions in which creative techniques were used to draw the viewer's attention. Second, this project models "symbolism": through analyses of semantic relationships between concrete objects and abstract concepts, the system aims to decode symbolic associations that humans make. The ability to automatically understand vividness and symbolism is key to building computational intelligence that can make inferences about what the media implies. This interdisciplinary project also has an educational component of potentially increasing the media literacy of school students, and involving college students from diverse backgrounds into computational research. The work can be used to discover patterns in how the visual rhetoric in the media evolved over time or how it differs in different cultures.This research pursues three directions. First, a framework for judging vividness (i.e., to what degree an image as a whole is vivid; what part of an image is vivid; and whether a text snippet is vivid) is developed. Data about the vividness of a variety of images and text is collected from human annotators. Cues and techniques such as saliency, attention, sentiment, memorability and abnormality are used to build prediction models for vividness. Second, two pipelines for detecting symbolic references are developed. One pipeline hypothesizes potential signifiers from an image, then uses textual resources to map these to signifieds. The other pipeline directly hypothesizes what the signifieds might be, and obtains training data for these from web resources. The outputs from these pipelines are combined to generate the signifier-signified pairs. Third, a method for generating explanations of the strategies is developed, using the vividness and symbolism outputs. Numerous resources to be shared with the research community are developed over the course of the project.