MMT: Multi-way Multi-modal Transformer for Multimodal Learning
MMT: Multi-way Multi-modal Transformer for Multimodal Learning
复制标题
DOI：
10.24963/ijcai.2022/480
复制
发表时间：
2022-07
期刊：
影响因子：
--
通讯作者：
Jiajia Tang;Kang Li;Ming Hou;Xuanyu Jin;Wanzeng Kong;Yu Ding;Qianchuan Zhao
中科院分区：
文献类型：
--
作者：
Jiajia Tang;Kang Li;Ming Hou;Xuanyu Jin;Wanzeng Kong;Yu Ding;Qianchuan Zhao
多模式学习研究的核心是有效利用多种模态融合表示形式的挑战。但是，现有的双向跨模式单向关注只能利用从一个源到一个目标模态的模式间相互作用。确实，这确实无法释放多模式融合的完整表达能力，并具有限制数量的模态和固定的交互式方向。在这项工作中，提出了多道路的多模式变压器（MMT），以同时探索通过单个块而不是单个模式的多道路多模式的多道路多模式的多模式堆叠的交叉模式块。 MMT的核心思想是多道路多模式的关注，在其中利用多种方式来计算多路注意张量。这自然会使我们利用综合的多模式互动路径。具体而言，多路张量由由多个互连模态感知的核心张量组成，这些核心张量包括由Amodal内部相互作用组成。此外，张量收缩操作用于研究不同核心张量之间的模式依赖性。实际上，我们的基于张量的多路结构可以轻松地将MMT扩展到与任意数量模态相关的情况下。以MMT为基础，进一步建立了层次网络，以递归将低级多通道多模式相互作用传输到高级相互作用。该实验表明，MMT可以实现最先进或可比的性能。
The heart of multimodal learning research lies the challenge of effectively exploiting fusion representations among multiple modalities.However, existing two-way cross-modality unidirectional attention could only exploit the intermodal interactions from one source to one target modality. This indeed fails to unleash the complete expressive power of multimodal fusion with restricted number of modalities and fixed interactive direction.In this work, the multiway multimodal transformer (MMT) is proposed to simultaneously explore multiway multimodal intercorrelations for each modality via single block rather than multiple stacked cross-modality blocks. The core idea of MMT is the multiway multimodal attention, where the multiple modalities are leveraged to compute the multiway attention tensor. This naturally benefits us to exploit comprehensive many-to-many multimodal interactive paths. Specifically, the multiway tensor is comprised of multiple interconnected modality-aware core tensors that consist of the intramodal interactions. Additionally, the tensor contraction operation is utilized to investigate intermodal dependencies between distinct core tensors.Essentially, our tensor-based multiway structure allows for easily extending MMT to the case associated with an arbitrary number of modalities. Taking MMT as the basis, the hierarchical network is further established to recursively transmit the low-level multiway multimodal interactions to high-level ones. The experiments demonstrate that MMT can achieve state-of-the-art or comparable performance.