TPFN: Applying Outer Product Along Time to Multimodal Sentiment Analysis Fusion on Incomplete Data
TPFN: Applying Outer Product Along Time to Multimodal Sentiment Analysis Fusion on Incomplete Data
复制标题
DOI：
10.1007/978-3-030-58586-0_26
复制
发表时间：
2020
期刊：
影响因子：
--
通讯作者：
Binghua Li;Chao Li;Feng Duan;Ning Zheng;Qibin Zhao
中科院分区：
文献类型：
--
作者：
Binghua Li;Chao Li;Feng Duan;Ning Zheng;Qibin Zhao
多通道情感分析在计算机视觉和自然语言处理中都得到了广泛的研究。然而，尽管现实世界中普遍存在这样的问题，但对不完美数据，特别是具有缺失值的数据的研究，仍然远未取得成功和具有挑战性。虽然以前的工作通过利用融合特征的低阶结构显示了良好的性能，但只涉及到时间动力学的一阶统计量。为此，我们提出了一种新的网络结构，称为时间乘积融合网络(TPFN)，它同时考虑了高阶统计量和时间动力学。我们通过沿相邻时间步长的外积构造融合特征，从而利用更丰富的模态和时间交互。此外，我们认为可以通过对潜在因子的Frobenius范数进行正则化来获得低阶结构，而不是通过融合特征来获得。在CMU-MOSI和CMU-MOSEI数据集上的实验表明，在随机和结构化缺失值的情况下，TPFN在多模式情感分析方面都能与最新的方法竞争。
Multimodal sentiment analysis (MSA) has been widely investigated in both computer vision and natural language processing. However, studies on the imperfect data especially with missing values are still far from success and challenging, even though such an issue is ubiquitous in the real world. Although previous works show the promising performance by exploiting the low-rank structures of the fused features, only the first-order statistics of the temporal dynamics are concerned. To this end, we propose a novel network architecture termed Time Product Fusion Network (TPFN), whichtakes the high-order statistics over both modalities and temporal dynamics into account. We construct the fused features by the outer product along adjacent time-steps, such that richer modal and temporal interactions are utilized. In addition, we claim that the low-rank structures can be obtained by regularizing the Frobenius norm of latent factors instead of the fused features. Experiments on CMU-MOSI and CMU-MOSEI datasets show that TPFN can compete with state-of-the art approaches in multimodal sentiment analysis in cases of both random and structured missing values.