JECL: Joint Embedding and Cluster Learning for Image-Text Pairs
JECL: Joint Embedding and Cluster Learning for Image-Text Pairs
复制标题
DOI：
10.1109/icpr48806.2021.9412667
复制
发表时间：
2019-01
期刊：
影响因子：
--
通讯作者：
Sean T. Yang;Kuan-Hao Huang;Bill Howe
中科院分区：
文献类型：
--
作者：
Sean T. Yang;Kuan-Hao Huang;Bill Howe
我们提出了 JECL，一种通过训练具有正则化聚类和对齐目标的并行编码器来聚类图像标题对的方法，同时学习表示和聚类分配。这些图像标题对经常出现在高价值应用中，在这些应用中，生成结构化训练数据的成本很高，但自由文本描述很常见。 JECL 通过最小化图像和文本分布之间的 Kullback-Leibler 散度与组合联合目标分布之间的散度并优化图像和文本的软聚类分配之间的 Jensen-Shannon 散度来进行训练。正则化器也应用于 JECL，以防止出现琐碎的解决方案。实验表明，JECL 在大型基准图像字幕数据集上的性能优于单视图和多视图方法，并且对于丢失字幕和变化的数据大小具有非常鲁棒性。
We propose JECL, a method for clustering image-caption pairs by training parallel encoders with regularized clustering and alignment objectives, simultaneously learning both representations and cluster assignments. These image-caption pairs arise frequently in high-value applications where structured training data is expensive to produce, but free-text descriptions are common. JECL trains by minimizing the Kullback-Leibler divergence between the distribution of the images and text to that of a combined joint target distribution and optimizing the Jensen-Shannon divergence between the soft cluster assignments of the images and text. Regularizers are also applied to JECL to prevent trivial solutions. Experiments show that JECL outperforms both single-view and multi-view methods on large benchmark image-caption datasets, and is remarkably robust to missing captions and varying data sizes.