Cross-Modal and Hierarchical Modeling of Video and Text
Cross-Modal and Hierarchical Modeling of Video and Text
复制标题
DOI：
10.1007/978-3-030-01261-8_23
复制
发表时间：
2018-09
期刊：
影响因子：
--
通讯作者：
Bowen Zhang;Hexiang Hu;Fei Sha
中科院分区：
文献类型：
--
作者：
Bowen Zhang;Hexiang Hu;Fei Sha
视觉数据和文本数据是由多粒度的信息组成的。视频可以描述由多个片段或镜头组成的复杂场景，其中每个片段或镜头描述语义上连贯的事件或动作。同样，一个段落可能包含不同主题的句子，这些句子共同传达了一个连贯的信息或故事。在本文中，我们调查的建模技术，这样的层次顺序数据，其中有多个模态之间的对应关系。具体来说，我们引入层次序列嵌入（HSE），一个通用的模型，用于将不同模态的序列数据嵌入到层次语义空间中，具有显式或隐式的对应信息。我们在大规模视频和段落检索数据集上进行了实证研究，并证明了所提出的方法具有上级性能。此外，我们研究了我们学到的嵌入应用于下游任务时的有效性。我们展示了它在零镜头动作识别和视频字幕中的实用性。
Visual data and text data are composed of information at multiple granularities. A video can describe a complex scene that is composed of multiple clips or shots, where each depicts a semantically coherent event or action. Similarly, a paragraph may contain sentences with different topics, which collectively conveys a coherent message or story. In this paper, we investigate the modeling techniques for such hierarchical sequential data where there are correspondences across multiple modalities. Specifically, we introduce hierarchical sequence embedding (HSE), a generic model for embedding sequential data of different modalities into hierarchically semantic spaces, with either explicit or implicit correspondence information. We perform empirical studies on large-scale video and paragraph retrieval datasets and demonstrated superior performance by the proposed methods. Furthermore, we examine the effectiveness of our learned embeddings when applied to downstream tasks. We show its utility in zero-shot action recognition and video captioning.