A Cross-Modal Approach for Extracting Semantic Relationships Between Concepts Using Tagged Images
A Cross-Modal Approach for Extracting Semantic Relationships Between Concepts Using Tagged Images
复制标题
DOI：
10.1109/tmm.2014.2306655
复制
发表时间：
2014-06
影响因子：
7.3
通讯作者：
Marie Katsurai;Takahiro Ogawa;M. Haseyama
中科院分区：
文献类型：
--
作者：
Marie Katsurai;Takahiro Ogawa;M. Haseyama
本文提出了一种跨模态的方法提取概念之间的语义关系，使用标记的图像。在所提出的方法中，我们首先使用典型相关分析（CCA）将标记图像的文本和视觉特征投影到潜在空间。然后，在CCA的概率解释下，我们计算每个概念的潜在变量的代表性分布。基于概念的代表性分布，我们推导出两种类型的度量：概念之间的语义相关性和每个概念的抽象程度。由于这些措施是来自一个跨模态的计划，使文本和视觉功能的协同使用，语义关系可以成功地反映语义和视觉环境。从Flickr上收集的标记图像进行的实验表明，我们的措施是更连贯的人类认知比传统的措施，使用文本或视觉功能，或基于WordNet的措施。特别是，一种新的语义相关性度量，它满足三角不等式，在我们的框架中不同的距离测量中获得最好的结果。我们的措施的多媒体相关的任务，如概念聚类，图像注释和标签推荐的适用性也显示在实验中。
This paper presents a cross-modal approach for extracting semantic relationships between concepts using tagged images. In the proposed method, we first project both text and visual features of the tagged images to a latent space using canonical correlation analysis (CCA). Then, under the probabilistic interpretation of CCA, we calculate a representative distribution of the latent variables for each concept. Based on the representative distributions of the concepts, we derive two types of measures: the semantic relatedness between the concepts and the abstraction level of each concept. Because these measures are derived from a cross-modal scheme that enables the collaborative use of both text and visual features, the semantic relationships can successfully reflect semantic and visual contexts. Experiments conducted on tagged images collected from Flickr show that our measures are more coherent to human cognition than the conventional measures that use either text or visual features, or the WordNet-based measures. In particular, a new measure of semantic relatedness, which satisfies the triangle inequality, obtains the best results among different distance measures in our framework. The applicability of our measures to multimedia-related tasks such as concept clustering, image annotation and tag recommendation is also shown in the experiments.