Combining Multiple Cues for Visual Madlibs Question Answering
Combining Multiple Cues for Visual Madlibs Question Answering
复制标题
DOI：
10.1007/s11263-018-1096-0
复制
发表时间：
2016-11
影响因子：
19.5
通讯作者：
T. Tommasi;Arun Mallya;Bryan A. Plummer;Svetlana Lazebnik;A. Berg;Tamara L. Berg
中科院分区：
文献类型：
--
作者：
T. Tommasi;Arun Mallya;Bryan A. Plummer;Svetlana Lazebnik;A. Berg;Tamara L. Berg
本文提出了一种基于Visual Madlibs数据集的填空式选择题的求解方法。我们的方法不是在ImageNet分类任务上训练的通用和常用的表示，而是采用了针对场景识别、人员活动分类和属性预测等专门任务训练的网络组合。我们还提出了一种方法，从候选答案定位短语，以提供空间支持的特征提取。我们通过归一化典型相关分析（nCCA）将这些特征中的每一个与候选答案一起映射到联合嵌入空间。最后，我们解决了一个优化问题，学习联合收割机的分数从多个线索训练nCCA模型选择最佳答案。广泛的实验结果表明，与之前的技术水平相比，它有了显着的改进，并证实回答各种类型的问题都受益于检查各种图像线索并仔细选择特征提取的空间支持。
This paper presents an approach for answering fill-in-the-blank multiple choice questions from the Visual Madlibs dataset. Instead of generic and commonly used representations trained on the ImageNet classification task, our approach employs a combination of networks trained for specialized tasks such as scene recognition, person activity classification, and attribute prediction. We also present a method for localizing phrases from candidate answers in order to provide spatial support for feature extraction. We map each of these features, together with candidate answers, to a joint embedding space through normalized canonical correlation analysis (nCCA). Finally, we solve an optimization problem to learn to combine scores from nCCA models trained on multiple cues to select the best answer. Extensive experimental results show a significant improvement over the previous state of the art and confirm that answering questions from a wide range of types benefits from examining a variety of image cues and carefully choosing the spatial support for feature extraction.