RI: Small: Modeling Vividness and Symbolism for Decoding Visual Rhetoric

该项目开发了分析和推断通过有说服力的图像和文本在媒体中传达的非文字信息的系统。设计了两种说服性策略的计算表示，以模拟可观察信息和潜在信息之间的映射。首先，这个项目建立了“生动”的模型：通过分析人类主体如何感知图像和文本，该系统旨在识别相关区域，在这些区域中，使用了创造性的技术来吸引观众的注意力。其次，这个项目建立了“象征主义”的模型：通过分析具体物体和抽象概念之间的语义关系，该系统旨在破译人类的符号联想。自动理解生动和象征意义的能力是构建计算智能的关键，计算智能可以对媒体所暗示的内容做出推断。这个跨学科的项目还有一个潜在的教育组成部分，那就是提高在校学生的媒体素养，并让来自不同背景的大学生参与计算研究。这项工作可以用来发现媒体中的视觉修辞如何随着时间的推移而演变的模式，或者它在不同文化中的差异。本研究追求三个方向。首先，开发了一个判断生动程度的框架(即，图像作为一个整体在多大程度上是生动的；图像的哪个部分是生动的；以及文本片段是否生动)。关于各种图像和文本的生动程度的数据是从人类注释员那里收集的。使用显著、注意、情绪、记忆力和异常等线索和技术来建立生动程度的预测模型。其次，开发了两条检测符号引用的流水线。一条管道假设来自图像的潜在能指，然后使用文本资源将这些映射到所指。另一个渠道直接假设这些重要信息可能是什么，并从网络资源中获取这些信息的训练数据。来自这些管道的输出被组合以产生能指-所指对。第三，提出了一种利用生动形象和象征意义的输出生成策略解释的方法。在该项目的整个过程中，开发了许多与研究界共享的资源。

This project develops systems for analyzing and inferring the non-literal messages conveyed in the media through persuasive images and text. Computational representations of two persuasive strategies are devised to model the mapping between observable information and underlying messages. First, this project models "vividness": through analyses of how human subjects perceive images and text, the system aims to identify relevant regions in which creative techniques were used to draw the viewer's attention. Second, this project models "symbolism": through analyses of semantic relationships between concrete objects and abstract concepts, the system aims to decode symbolic associations that humans make. The ability to automatically understand vividness and symbolism is key to building computational intelligence that can make inferences about what the media implies. This interdisciplinary project also has an educational component of potentially increasing the media literacy of school students, and involving college students from diverse backgrounds into computational research. The work can be used to discover patterns in how the visual rhetoric in the media evolved over time or how it differs in different cultures.This research pursues three directions. First, a framework for judging vividness (i.e., to what degree an image as a whole is vivid; what part of an image is vivid; and whether a text snippet is vivid) is developed. Data about the vividness of a variety of images and text is collected from human annotators. Cues and techniques such as saliency, attention, sentiment, memorability and abnormality are used to build prediction models for vividness. Second, two pipelines for detecting symbolic references are developed. One pipeline hypothesizes potential signifiers from an image, then uses textual resources to map these to signifieds. The other pipeline directly hypothesizes what the signifieds might be, and obtains training data for these from web resources. The outputs from these pipelines are combined to generate the signifier-signified pairs. Third, a method for generating explanations of the strategies is developed, using the vividness and symbolism outputs. Numerous resources to be shared with the research community are developed over the course of the project.