Semi-Supervised Multi-Modal Clustering and Classification with Incomplete Modalities
Semi-Supervised Multi-Modal Clustering and Classification with Incomplete Modalities
复制标题
DOI：
10.1109/tkde.2019.2932742
复制
发表时间：
2021-02
影响因子：
8.9
通讯作者：
Yang Yang-Yang;De-chuan Zhan;Yi-Feng Wu;Zhi-Bin Liu;Hui Xiong;Yuan Jiang
中科院分区：
文献类型：
--
作者：
Yang Yang-Yang;De-chuan Zhan;Yi-Feng Wu;Zhi-Bin Liu;Hui Xiong;Yuan Jiang
本文提出了一种新的同时考虑模态一致性和互补性的不完全模态半监督学习（SLIM）方法，以及基于矩阵完备化的核SLIM（SLIM-K）方法，进一步解决了模态不完全性问题。众所周知，大多数真实数据都具有多模态表示，多模态学习是指学习一个精确的模型来获得完整模态的过程。然而，由于数据采集的失败、自身的缺陷或其他各种原因，多模态实例通常具有不完整的模态，这就产生了使用现有方法的效用障碍。本文提出的SLIM算法综合了内在一致性和外在互补信息，同时进行预测和聚类。具体地说，SLIM在一个统一的框架下形成不同的模态分类器和聚类学习器，同时利用未标记数据的外部互补信息来克服模态不完整带来的不一致性。此外，为了从本质上解决模态缺失问题，我们提出了SLIM-K算法，将补核矩阵分别引入分类器和聚类学习器。因此，SLIM-K可以解决结果中缺少模态的缺陷。最后，我们讨论了不完备模态的推广问题。在13个基准多模态数据集和两个真实不完全多模态数据集上的实验验证了本文方法的有效性。
In this paper, we propose a novel Semi-supervised Learning with Incomplete Modality (SLIM) method considering the modal consistency and complementarity simultaneously, and Kernel SLIM (SLIM-K) based on matrix completion for further solving the modal incompleteness. As is well known, most realistic data have multi-modal representations, multi-modal learning refers to the process of learning a precise model for complete modalities. However, due to the failures of data collection, self-deficiencies, or other various reasons, multi-modal examples are usually with incomplete modalities, which generate utility obstacle using previous methods. In this paper, SLIM integrates the intrinsic consistency and extrinsic complementary information for prediction and cluster simultaneously. In detail, SLIM forms different modal classifiers and clustering learner consistently in a unified framework, while using the extrinsic complementary information from unlabeled data against the insufficiencies brought by the incomplete modal issue. Moreover, in order to deal with missing modality in essence, we propose the SLIM-K, which takes the complemented kernel matrix into the classifiers and the cluster learner respectively. Thus, SLIM-K can solve the defects of missing modality in result. Finally, we give the discussion of generalization of incomplete modalities. Experiments on 13 benchmark multi-modal datasets and two real-world incomplete multi-modal datasets validate the effectiveness of our methods.