Multimodal Neural Machine Translation Using CNN and Transformer Encoder
Multimodal Neural Machine Translation Using CNN and Transformer Encoder
复制标题
DOI：
10.29007/hxhn
复制
发表时间：
2019-04
期刊：
影响因子：
--
通讯作者：
Hiroki Takushima;Akihiro Tamura;Takashi Ninomiya;Hideki Nakayama
中科院分区：
文献类型：
--
作者：
Hiroki Takushima;Akihiro Tamura;Takashi Ninomiya;Hideki Nakayama
多模态机器翻译使用与源语言句子相关的图像作为输入来提高翻译质量。以往的多模态神经机器翻译（NMT）模型将图像区域的视觉特征整合到源语言句子的编码器中或编码器和解码器之间的注意机制中，无法捕捉图像区域视觉特征之间的关系。本文提出了一种新的多模态NMT模型，它使用卷积神经网络（CNN）和Transformer编码器对输入图像进行编码。特别地，所提出的图像编码器首先使用CNN从每个图像区域提取视觉特征，然后使用Transformer编码器基于所提取的视觉特征对输入图像进行编码，其中来自每个图像区域的视觉特征之间的关系由Transformer编码器的自注意机制捕获。使用Multi 30 k数据集进行的英语-德语翻译任务的实验表明，该模型相对于没有图像输入的基线Transformer NMT模型实现了0.96 BLEU点的改进，并且相对于没有图像的Transformer编码器的基线多模态Transformer NMT模型实现了0.47 BLEU点的改进。
Multimodal machine translation uses images related to source language sentences as inputs to improve translation quality. Previous multimodal Neural Machine Translation (NMT) models, which incorporate visual features of each image region into an encoder for source language sentences or an attention mechanism between an encoder and a decoder, cannot catch the relation between visual features from each image region. This paper proposes a new multimodal NMT model, which encodes an input image using a Convolutional Neural Network (CNN) and a Transformer encoder. In particular, the proposed image encoder first extracts visual features from each image region using a CNN, and then encodes an input image on the basis of the extracted visual features using a Transformer encoder, where the relation between visual features from each image region are captured by a self-attention mechanism of the Transformer encoder. The experiments on the English-German translation task using the Multi30k data set show that the proposed model achieves 0.96 BLEU points improvement against a baseline Transformer NMT model without image inputs and 0.47 BLEU points improvement against a baseline multimodal Transformer NMT model without a Transformer encoder for images.