Quantifying the Visual Concreteness of Words and Topics in Multimodal Datasets
Quantifying the Visual Concreteness of Words and Topics in Multimodal Datasets
复制标题
DOI：
10.18653/v1/n18-1199
复制
发表时间：
2018-04
期刊：
影响因子：
--
通讯作者：
Jack Hessel;David Mimno;Lillian Lee
中科院分区：
文献类型：
--
作者：
Jack Hessel;David Mimno;Lillian Lee
多模态机器学习算法旨在学习视觉-文本对应关系。以前的工作表明，具体的视觉表现形式的概念可能比抽象的概念更容易学习。我们给出了一个自动计算多模态数据集内的单词和主题的视觉具体性的算法。我们在四种设置中应用该方法，从图像标题到从历史书籍中抓取的图像/文本。除了能够在多模态数据集中探索概念之外，我们的具体性分数还预测了机器学习算法学习文本/视觉关系的能力。我们发现：1）具体的概念确实更容易学习; 2）我们考虑的大量算法都有类似的失败案例; 3）具体性和性能之间的精确正相关关系在不同的数据集之间存在差异。最后，我们建议使用具体分数，以促进未来的多模态研究。
Multimodal machine learning algorithms aim to learn visual-textual correspondences. Previous work suggests that concepts with concrete visual manifestations may be easier to learn than concepts with abstract ones. We give an algorithm for automatically computing the visual concreteness of words and topics within multimodal datasets. We apply the approach in four settings, ranging from image captions to images/text scraped from historical books. In addition to enabling explorations of concepts in multimodal datasets, our concreteness scores predict the capacity of machine learning algorithms to learn textual/visual relationships. We find that 1) concrete concepts are indeed easier to learn; 2) the large number of algorithms we consider have similar failure cases; 3) the precise positive relationship between concreteness and performance varies between datasets. We conclude with recommendations for using concreteness scores to facilitate future multimodal research.