CRII: RI: Automatically Understanding the Messages and Goals of Visual Media

这个项目开发了解释图像的视觉修辞的技术。该项目通过对解码广告和艺术照片中的视觉信息这一新颖问题的新颖解决方案来推动计算机视觉，从而使计算机视觉更接近其能够自动理解视觉内容的目标。从实践的角度来看，理解视觉修辞可以用来为视障人士制作符合人类对这些图像的标签的图像描述，从而使他们能够接触到报纸或电视上显示的丰富内容。该项目与教育紧密结合。这项工作是跨学科的，可以吸引来自不同领域的本科生参与研究。本研究侧重于三个媒体理解任务：(1)理解艺术图像所传达的有说服力的信息，以及这些图像用来传达其信息的策略；(2)揭露摄影师对其主题的偏见，例如，确定一张照片是以积极的还是消极的光描绘其主题；以及(3)预测艺术照片的哪一部分最吸引人或最令人心酸。为了能够对艺术图像进行解码，收集了大量的数据集，并用一些旨在供人类理解的艺术属性和说服技术进行标注，然后开发了在艺术图像中建模视觉象征的方法，以及从情感分析中采用正/负效应方法。为了预测摄影师对拍摄对象的偏见，收集了少数民族和外国人的历史和现代肖像的数据集，然后创建了一个算法，根据照片的肢体语言和3D布局和构图进行推理。为了预测疼痛，收集了一组著名摄影师的艺术图像的眼动数据，然后分析了照片中对象之间的语义和内涵冲突。

This project develops technologies to interpret the visual rhetoric of images. The project advances computer vision through novel solutions to the novel problem of decoding the visual messages in advertisements and artistic photographs, and thus brings computer vision closer to its goal of being able to automatically understand visual content. From a practical standpoint, understanding visual rhetoric can be used to produce image descriptions for the visually impaired that align with how a human would label these images, and thus give them access to the rich content shown in newspapers or on TV. This project is tightly integrated with education. The work is interdisciplinary and can attract undergraduate students to the research from different fields. This research focuses on three media understanding tasks: (1) understanding the persuasive messages conveyed by artistic images and the strategies that those images use to convey their message; (2) exposing a photographer's bias towards their subject, e.g., determining whether a photograph portrays its subject in a positive or negative light; and (3) predicting what part of an artistic photograph a viewer might find most captivating or poignant. To enable decoding of artistic images, a large dataset is collected and annotated with a number of artistic properties and persuasion techniques that are intended for human understanding, then methods are developed to model visual symbolism in artistic images, as well as adapt positive/negative effect methods from sentiment analysis. To predict the photographer's bias towards a subject, a dataset of historical and modern portrayals of minorities and foreigners is collected, then an algorithm is created that reasons about body language and 3D layout and composition of the photo. To predict poignance, eyetracking data on a set of artistic images from famous photographers is collected, then semantic and connotation conflicts between the objects in the photographs are analyzed.