Learning Language-Visual Embedding for Movie Understanding with Natural-Language
Learning Language-Visual Embedding for Movie Understanding with Natural-Language
复制标题
DOI：
--
复制
发表时间：
2016-09
期刊：
影响因子：
--
通讯作者：
Atousa Torabi;Niket Tandon;L. Sigal
中科院分区：
文献类型：
--
作者：
Atousa Torabi;Niket Tandon;L. Sigal
学习联合语言-视觉嵌入具有许多非常吸引人的特性，并且可以导致各种实际应用，包括自然语言图像/视频注释和搜索。在这项工作中，我们研究了三种不同的联合语言视觉神经网络模型架构。我们在大规模LSMDC 16电影数据集上评估了我们的模型，用于两个任务：1）视频注释和检索的标准排名2）我们提出的电影多项选择测试。该测试有助于基于人类活动的自然语言视频注释的视觉语言模型的自动评估。除了作为LSMDC 16的一部分提供的原始音频描述（AD）字幕外，我们还收集并将提供a）使用Amazon MTurk手动生成的字幕改写B）基于“Knowlywood”（一种活动知识挖掘模型）自动生成的“谓词+对象”（PO）短语中的人类活动元素。我们最好的模型在1000个样本的子集上，在注释上的Recall@10为19.2%，在视频检索任务上的Recall@10为18.9%。对于多项选择测试，我们的最佳模型在整个LSMDC 16公共测试集上的准确率达到58.11%。
Learning a joint language-visual embedding has a number of very appealing properties and can result in variety of practical application, including natural language image/video annotation and search. In this work, we study three different joint language-visual neural network model architectures. We evaluate our models on large scale LSMDC16 movie dataset for two tasks: 1) Standard Ranking for video annotation and retrieval 2) Our proposed movie multiple-choice test. This test facilitate automatic evaluation of visual-language models for natural language video annotation based on human activities. In addition to original Audio Description (AD) captions, provided as part of LSMDC16, we collected and will make available a) manually generated re-phrasings of those captions obtained using Amazon MTurk b) automatically generated human activity elements in "Predicate + Object" (PO) phrases based on "Knowlywood", an activity knowledge mining model. Our best model archives Recall@10 of 19.2% on annotation and 18.9% on video retrieval tasks for subset of 1000 samples. For multiple-choice test, our best model achieve accuracy 58.11% over whole LSMDC16 public test-set.