Leveraging multimodal information for event summarization and concept-level sentiment analysis
Leveraging multimodal information for event summarization and concept-level sentiment analysis
复制标题
DOI：
10.1016/j.knosys.2016.05.022
复制
发表时间：
2016-09
期刊：
影响因子：
--
通讯作者：
R. Shah;Yi Yu;Akshay Verma;Suhua Tang;A. Shaikh;Roger Zimmermann
中科院分区：
文献类型：
--
作者：
R. Shah;Yi Yu;Akshay Verma;Suhua Tang;A. Shaikh;Roger Zimmermann
在线用户生成内容（UGC）数量的快速增长需要社交媒体公司从照片和视频中自动提取知识结构（概念），以提供多样化的多媒体相关服务。然而，现实世界的照片和视频是复杂的和嘈杂的，并且单独从多媒体内容中提取语义和感觉是一项非常困难的任务，因为合适的概念可能以不同的表示来展示。因此，为了更好地理解，需要从多个模态分析UGC。为此，我们首先提出了EventBuilder系统，它处理语义理解，并自动生成一个给定的事件的多媒体摘要，实时利用不同的社交媒体，如维基百科和Flickr。随后，我们提出了EventSensor系统，旨在解决感官理解，并产生一个给定的情绪的多媒体摘要。它从UGC的视觉内容和文本元数据中提取概念和情绪标签，并利用它们来支持几个重要的多媒体相关服务，如音乐多媒体摘要。此外，EventSensor通过利用EventBuilder作为其语义引擎组件，支持基于语义的事件摘要。实验结果表明，EventBuilder和EventSensor都优于其基线，并有效地总结了YFCC 100M数据集上的知识结构。
The rapid growth in the amount of user-generated content (UGCs) online necessitates for social media companies to automatically extract knowledge structures (concepts) from photos and videos to provide diverse multimedia-related services. However, real-world photos and videos are complex and noisy, and extracting semantics and sentics from the multimedia content alone is a very difficult task because suitable concepts may be exhibited in different representations. Hence, it is desirable to analyze UGCs from multiple modalities for a better understanding. To this end, we first present the EventBuilder system that deals with semantics understanding and automatically generates a multimedia summary for a given event in real-time by leveraging different social media such as Wikipedia and Flickr. Subsequently, we present the EventSensor system that aims to address sentics understanding and produces a multimedia summary for a given mood. It extracts concepts and mood tags from visual content and textual metadata of UGCs, and exploits them in supporting several significant multimedia-related services such as a musical multimedia summary. Moreover, EventSensor supports sentics-based event summarization by leveraging EventBuilder as its semantics engine component. Experimental results confirm that both EventBuilder and EventSensor outperform their baselines and efficiently summarize knowledge structures on the YFCC100M dataset.