Concept-Level Multimodal Ranking of Flickr Photo Tags via Recall Based Weighting
Concept-Level Multimodal Ranking of Flickr Photo Tags via Recall Based Weighting
复制标题
DOI：
10.1145/2983554.2983555
复制
发表时间：
2016-10
期刊：
影响因子：
--
通讯作者：
R. Shah;Yi Yu;Suhua Tang;S. Satoh;Akshay Verma;Roger Zimmermann
中科院分区：
文献类型：
--
作者：
R. Shah;Yi Yu;Suhua Tang;S. Satoh;Akshay Verma;Roger Zimmermann
社交媒体平台允许用户使用标签来注释照片，这些标签显著促进了对照片的有效语义理解、搜索和检索。然而，由于用户标记的手动、模糊和个性化的性质，照片的许多标记是随机顺序的，甚至与视觉内容无关。为了自动计算标签相关性为一个给定的照片，我们提出了一个标签排名计划的基础上投票的照片邻居来自多模态信息。具体来说，我们确定照片邻居利用地理，视觉和语义概念来自空间信息，视觉内容和文本元数据，分别。我们利用高层次的功能，而不是传统的低层次的功能来计算标签的相关性。对来自YFCC 100M数据集的203，840张照片的代表性集的实验结果证实了上述多模态概念在计算标签相关性方面相互补充。此外，我们探索多模态信息的融合，以完善标签排名利用召回为基础的加权。在代表集上的实验结果证实了该算法的性能优于现有技术。
Social media platforms allow users to annotate photos with tags that significantly facilitate an effective semantics understanding, search, and retrieval of photos. However, due to the manual, ambiguous, and personalized nature of user tagging, many tags of a photo are in a random order and even irrelevant to the visual content. Aiming to automatically compute tag relevance for a given photo, we propose a tag ranking scheme based on voting from photo neighbors derived from multimodal information. Specifically, we determine photo neighbors leveraging geo, visual, and semantics concepts derived from spatial information, visual content, and textual metadata, respectively. We leverage high-level features instead traditional low-level features to compute tag relevance. Experimental results on a representative set of 203,840 photos from the YFCC100M dataset confirm that above-mentioned multimodal concepts complement each other in computing tag relevance. Moreover, we explore the fusion of multimodal information to refine tag ranking leveraging recall based weighting. Experimental results on the representative set confirm that the proposed algorithm outperforms state-of-the-arts.