CRII: RI: Automatically Understanding the Messages and Goals of Visual Media

CRII：RI：自动理解视觉媒体的信息和目标

• 批准号: 1566270
• 负责人: Adriana Kovashka
• 金额: $ 17.46万
• 依托单位: University of Pittsburgh
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2016
• 资助国家: 美国
• 起止时间: 2016-06-01 至 2019-05-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1566270&HistoricalAwards=false
• 关键词: CRII; RI; Automatically; Understanding; Messages

This project develops technologies to interpret the visual rhetoric of images. The project advances computer vision through novel solutions to the novel problem of decoding the visual messages in advertisements and artistic photographs, and thus brings computer vision closer to its goal of being able to automatically understand visual content. From a practical standpoint, understanding visual rhetoric can be used to produce image descriptions for the visually impaired that align with how a human would label these images, and thus give them access to the rich content shown in newspapers or on TV. This project is tightly integrated with education. The work is interdisciplinary and can attract undergraduate students to the research from different fields. This research focuses on three media understanding tasks: (1) understanding the persuasive messages conveyed by artistic images and the strategies that those images use to convey their message; (2) exposing a photographer's bias towards their subject, e.g., determining whether a photograph portrays its subject in a positive or negative light; and (3) predicting what part of an artistic photograph a viewer might find most captivating or poignant. To enable decoding of artistic images, a large dataset is collected and annotated with a number of artistic properties and persuasion techniques that are intended for human understanding, then methods are developed to model visual symbolism in artistic images, as well as adapt positive/negative effect methods from sentiment analysis. To predict the photographer's bias towards a subject, a dataset of historical and modern portrayals of minorities and foreigners is collected, then an algorithm is created that reasons about body language and 3D layout and composition of the photo. To predict poignance, eyetracking data on a set of artistic images from famous photographers is collected, then semantic and connotation conflicts between the objects in the photographs are analyzed.

该项目开发技术来解释图像的视觉修辞。该项目通过对广告和艺术照片中的视觉信息进行解码的新问题的新解决方案来推进计算机视觉，从而使计算机视觉更接近其能够自动理解视觉内容的目标。从实践的角度来看，理解视觉修辞可以用来为视障人士制作图像描述，这些描述与人类如何标记这些图像相一致，从而使他们能够获得报纸或电视上显示的丰富内容。该项目与教育紧密结合。这项工作是跨学科的，可以吸引不同领域的本科生进行研究。本研究主要针对三个媒体理解任务：（1）理解艺术图像所传达的说服性信息以及这些图像用来传达信息的策略;（2）暴露摄影师对主题的偏见，例如，判断一张照片是以正面还是负面的光线描绘其主题;以及（3）预测观看者可能会发现艺术照片的哪一部分最迷人或最令人心酸。为了能够解码艺术图像，收集了一个大型数据集，并使用许多旨在用于人类理解的艺术属性和说服技术进行注释，然后开发了对艺术图像中的视觉符号进行建模的方法，以及从情感分析中调整积极/消极效果的方法。为了预测摄影师对主题的偏见，收集了少数民族和外国人的历史和现代肖像的数据集，然后创建了一个算法，可以对身体语言和3D布局以及照片的构图进行推理。为了预测辛酸，收集了一组著名摄影师的艺术图像的眼动数据，然后分析了照片中物体之间的语义和内涵冲突。

项目成果

Predicting Visual Political Bias Using Webly Supervised Data and an Auxiliary Task

• DOI: 10.1007/s11263-021-01506-3
• 发表时间: 2021-08
• 期刊: International Journal of Computer Vision
• 影响因子: 19.5
• 作者: Christopher Thomas;Adriana Kovashka

Learning to Overcome Noise in Weak Caption Supervision for Object Detection

• DOI: 10.1109/tpami.2022.3187350
• 发表时间: 2022-06
• 期刊: IEEE Transactions on Pattern Analysis and Machine Intelligence
• 影响因子: 23.6
• 作者: Mesut Erhan Unal;Keren Ye;Mingda Zhang;Christopher Thomas;Adriana Kovashka;Wei Li;Danfeng Qin