Deep-Structured Event Modeling for User-Generated Photos
Deep-Structured Event Modeling for User-Generated Photos
复制标题
DOI：
10.1109/tmm.2017.2788210
复制
发表时间：
2018-08
影响因子：
7.3
通讯作者：
Xiaoshan Yang;Tianzhu Zhang;Changsheng Xu
中科院分区：
文献类型：
--
作者：
Xiaoshan Yang;Tianzhu Zhang;Changsheng Xu
基于视觉的事件分析由于以下挑战而变得困难。第一个挑战是组内差异。用户上传的照片是随着时间的推移对事件的视觉外观进行稀疏采样。因此，每张照片可能仅捕获特定复杂事件的单个对象或场景。第二个挑战是阶级间的混乱。与不同事件相关的照片可能包含类似的对象或场景。第三，不寻常事件的特点是稀缺，只有少数样本可用于学习事件模式。在本文中，通过考虑照片的时间戳，我们提出了一个结构化的事件建模（SEM）框架的事件分析，利用时间信息的视觉特征和事件类的照片序列。具体而言，使用深度神经网络（卷积神经网络和递归神经网络）和条件随机场来联合学习照片序列的时间事件模式和不同照片的关系。我们在两个应用程序中评估所提出的SEM框架：多类事件识别和照片序列中的异常事件检测。在公共事件识别数据集和异常事件数据集上的实验结果表明了该方法的有效性。
Vision-based event analysis is difficult because of the following challenges. The first challenge is intraclass variation. Photos uploaded by users are sparsely sampled visual appearances of an event over time. Thus, each photo may only capture a single object or scene of a specific complex event. The second challenge is interclass confusion. Photos related to different events may contain similar objects or scenes. Third, unusual events are characterized by scarcity, and only a few samples are available for use in learning event patterns. In this paper, by considering the photo timestamp, we propose a structured event modeling (SEM) framework for event analysis that exploits the temporal information of visual features and event classes in a photo sequence. Specifically, the temporal event patterns of the photo sequence and the relationships of different photos are jointly learned using deep neural networks (convolutional neural networks and recurrent neural networks) and a conditional random field. We evaluate the proposed SEM framework in two applications: multiclass event recognition and unusual event detection in photo sequences. The results of extensive experiments performed on a public event recognition dataset and a collected unusual event dataset demonstrate the effectiveness of the proposed method.