Detection of Important Scenes in Baseball Videos via Bidirectional Time Lag Aware Deep Multiset Canonical Correlation Analysis
Detection of Important Scenes in Baseball Videos via Bidirectional Time Lag Aware Deep Multiset Canonical Correlation Analysis
复制标题
DOI：
10.1109/access.2021.3088284
复制
发表时间：
2021-01-01
期刊：
影响因子：
3.9
通讯作者：
Haseyama, Miki
中科院分区：
文献类型：
--
作者：
Hirasawa, Kaito;Maeda, Keisuke;Haseyama, Miki
本文介绍了一种基于双向时间滞后的异质模式之间的相关性最大化，用于检测棒球视频中重要场景的新方法。提出的方法可以通过使用棒球视频及其相应的推文来协作来检测重要场景。本文的技术贡献是双重的。首先，由于不仅“推文和相应多个以前的事件”之间存在时间滞后，还存在“事件和相应的多个以下推文”之间的时间段，该建议的方法考虑了这些双向时间滞后。具体而言，新引入了这种双向时间落后于其协方差矩阵推导的表示。其次，提出的方法采用了从推文和视频计算出的文本，视觉和音频功能，作为多模式时间序列功能。使用BITL-DMCCA预测的多模式特征，通过基于生成对抗网络的异常检测来检测重要的场景。提出的方法不需要带注释的任何培训数据。通过将提出的方法应用于实际棒球匹配获得的实验结果显示了所提出的方法的有效性。
A novel method for detection of important scenes in baseball videos based on correlation maximization between heterogeneous modalities via bidirectional time lag aware deep multiset canonical correlation analysis (BiTl-dMCCA) is presented in this paper. The proposed method enables detection of important scenes by collaboratively using baseball videos and their corresponding tweets. The technical contributions of this paper are twofold. First, since there are time lags between not only "tweets and corresponding multiple previous events" but also "events and corresponding multiple following posted tweets", the proposed method considers these bidirectional time lags. Specifically, the representation of such bidirectional time lags into the derivation of their covariance matrices is newly introduced. Second, the proposed method adopts textual, visual and audio features calculated from tweets and videos as multi-modal time series features. Important scenes are detected as abnormal scenes via anomaly detection based on a generative adversarial network using multi-modal features projected by BiTl-dMCCA. The proposed method does not need any training data with annotation. Experimental results obtained by applying the proposed method to actual baseball matches show the effectiveness of the proposed method.