Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing-猫眼课题宝

DOI：

10.1007/978-3-030-58580-8_26

发表时间：

2020-07

期刊：

ArXiv

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Yapeng Tian;Dingzeyu Li;Chenliang Xu

在本文中，我们引入了一个新的问题，命名为视听视频解析，其目的是将视频解析成时间事件片段，并将它们标记为可听，可见或两者兼而有之。这样的问题对于完整理解视频中描绘的场景至关重要。为了便于探索，我们收集了aLook，Listen，and Parse（LLP）数据集，以弱监督的方式研究视听视频解析。这个任务可以自然地表述为多模态多实例学习（MMIL）问题。具体而言，我们提出了一种新的混合注意力网络，同时探索单峰和跨模态的时间背景。我们开发了一个细心的MMIL池化方法，从不同的时间范围和方式自适应地探索有用的音频和视频内容。此外，我们发现和减轻模态偏见和嘈杂的标签问题与个人指导的学习机制和标签平滑技术，分别。实验结果表明，具有挑战性的视听视频解析，即使只有视频级的弱标签可以实现。我们提出的框架可以有效地利用单模态和跨模态的时间上下文，并减轻模态偏见和嘈杂的标签问题。

In this paper, we introduce a new problem, named audio-visual video parsing, which aims to parse a video into temporal event segments and label them as either audible, visible, or both. Such a problem is essential for a complete understanding of the scene depicted inside a video. To facilitate exploration, we collect aLook, Listen, and Parse(LLP) dataset to investigate audio-visual video parsing in a weakly-supervised manner. This task can be naturally formulated as a Multimodal Multiple Instance Learning (MMIL) problem. Concretely, we propose a novel hybrid attention network to explore unimodal and cross-modal temporal contexts simultaneously. We develop an attentive MMIL pooling method to adaptively explore useful audio and visual content from different temporal extent and modalities. Furthermore, we discover and mitigate modality bias and noisy label issues with an individual-guided learning mechanism and label smoothing technique, respectively. Experimental results show that the challenging audio-visual video parsing can be achieved even with only video-level weak labels. Our proposed framework can effectively leverage unimodal and cross-modal temporal contexts and alleviate modality bias and noisy labels problems.

Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing

文献摘要

相似文献