One-shot Video Graph Generation for Explainable Action Reasoning
One-shot Video Graph Generation for Explainable Action Reasoning
复制标题
用于可解释动作推理的一次性视频图生成
DOI：
10.1016/j.neucom.2022.02.069
复制
发表时间：
2022-06
期刊：
影响因子：
6
通讯作者：
Mohan Kankanhalli
中科院分区：
文献类型：
--
作者：
Yamin Han;Tao Zhuo;Peng Zhang;Wei Huang;Yufei Zha;Yanning Zhang;Mohan Kankanhalli
人类行为分析是理解不同视频内容的关键而具有挑战性的任务。最近，为了使视频动作的可解释推理，时空视频图结构被提出来表示视频状态变化的语义水平。然而，它需要对所有视频帧进行繁琐的手动注释，这是一个严重的限制。显然，如果视频图形生成过程可以自动化，则这种方法将具有极大扩展的适用性。提出了一种单镜头视频图（One-Shot Video Graph，OSVG）生成方法，该方法只需要对视频起始帧中的对象进行一次性标注，就可以实现更有效的可解释动作推理。我们首先估计预定义的相关对象的时间维度，通过采用一个建议的一次性目标感知跟踪策略。这有助于在所有视频帧中同时获得对象位置和链接对象。然后，每个视频帧的场景图可以由属性检测器和关系检测器基于估计的对象位置来构造。此外，为了进一步提高执行动作的推理准确性，设计了一种全连接条件随机场（CRF）的视频图形平滑机制。通过顺序地检查平滑视频图的每个状态转换（包括属性和关系），可以使用预定义的规则来识别发生的动作。在CAD-120++数据集和一个新收集的NTU RGBD++数据集上的实验表明，所提出的OSVG能够在状态识别和动作识别准确率方面优于其他最先进的视频动作推理策略。
Human action analysis is a critical yet challenging task for understanding diverse video content. Recently, to enable explainable reasoning of video actions, a spatio-temporal video graph structure was proposed to represent the video state changes at the semantic level. However, its requirement of tedious manual annotation of all the video frames is a serious limitation. Obviously, this approach would have a tremendously expanded applicability if the video graph generation process can be automated. In this paper, a One-Shot Video Graph (OSVG) generation approach is proposed for more effective explainable action reasoning, which only requires a one-time annotation of the objects in the starting frame of the video. We first estimate the predefined relevant objects across the temporal dimension by employing a proposed one-shot target-aware tracking strategy. This helps obtain the object locations and links objects simultaneously across all video frames. Then, the scene graph of each video frame can be constructed by an attribute detector and a relationship detector based on the estimated object locations. In addition, to further enhance the reasoning accuracy of performed actions, a video graph smoothing mechanism is designed with a fully-connected Conditional Random Field (CRF). By sequentially examining every state transition (including attributes and relationships) of the smoothed video graph, the actions occurring can be recognized using pre-defined rules. Experiments on the CAD-120++ dataset and a newly collected NTU RGBD++ dataset have verified that the proposed OSVG is able to outperform other state-of-the-art video action reasoning strategies on both state recognition and action recognition accuracy.
登录
查看更多内容
DOI：
10.1109/icassp.2019.8683464
发表时间：
2019-05
期刊：
ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
影响因子：
--
作者：
Hao Zhou;Chuanping Hu;Chongyang Zhang;Shengyang Shen
通讯作者：
Hao Zhou;Chuanping Hu;Chongyang Zhang;Shengyang Shen
DOI：
10.1007/978-3-642-16111-7_49
发表时间：
2010-09
期刊：
--
影响因子：
--
作者：
Joris IJsselmuiden;R. Stiefelhagen
通讯作者：
Joris IJsselmuiden;R. Stiefelhagen
影响因子：
7.3
作者：
Li, Dong;Yao, Ting;Rui, Yong
通讯作者：
Rui, Yong
DOI：
10.1109/iccv.2017.234
发表时间：
2017-02
期刊：
2017 IEEE International Conference on Computer Vision (ICCV)
影响因子：
--
作者：
Jean-Baptiste Alayrac;Josef Sivic;I. Laptev;Simon Lacoste-Julien
通讯作者：
Jean-Baptiste Alayrac;Josef Sivic;I. Laptev;Simon Lacoste-Julien
用于任意大小和长度视频中的动作识别的双流 3-D 卷积网络融合
影响因子：
7.3
作者：
Wang, Xuanhan;Gao, Lianli;Liu, Xianglong
通讯作者：
Liu, Xianglong