EAGER: Towards Human Centered Visual Understanding: Exploring the Intended and Interpreted Meaning of Images in Social Multimedia

该项目探索了计算机视觉的一个新方向，即对社交多媒体中与图像相关的上下文相关视觉语义进行建模。上下文相关的视觉语义，例如，社会多媒体中图像的意图和感知情感是基于与之相关的各种上下文信息动态形成的。这不同于传统计算机视觉研究集中研究的静态视觉语义，例如图像中呈现的对象类别。该项目开发了一套新的网络化和上下文感知的概率潜在语义模型，其将情境信息集成到视觉内容分析中，用于对情境相关的视觉语义进行建模。研究团队正在验证两个假设：1）上下文相关语义需要整体建模，并从相关图像集合中联合推断; 2）相关上下文相关视觉语义，如图像的预期和感知意义，也需要联合建模，以实现更强大的识别。该项目通过培训研究生和本科生与教育相结合。研究成果可应用于许多领域，如有针对性的在线广告;开源信息分析和社会事件预测;以及社会多媒体安全。

This project explores a new direction in computer vision, which is to model the context dependent visual semantics associated with images in social multimedia. The context dependent visual semantics, e.g., the intended and perceived sentiment of an image in social multimedia, are dynamically formed based on the various contextual information associated with it. This is different from the static visual semantics that conventional computer vision research focused on studying, such as the object category presented in the image.The project develops a set of new networked and context aware probabilistic latent semantic models, which integrate situated contextual information into visual content analysis for modeling context dependent visual semantics. The research team is verifying two hypotheses: 1) the context dependent semantics needs to be holistically modeled and jointly inferred from a collection of related images; and 2) related context dependent visual semantics, such as intended and perceived meaning of an image, also needs to be jointly modeled for more robust recognition.The project is integrated with education through training graduate and undergraduate students. The outcome of the research can be applied to many domains, such as targeted online advertisements; open source information analysis and social event prediction; and social multimedia security.