Leveraging Partial Dependency Trees to Control Image Captions
Leveraging Partial Dependency Trees to Control Image Captions
复制标题
DOI：
10.18653/v1/2021.alvr-1.3
复制
发表时间：
2021-06
期刊：
影响因子：
--
通讯作者：
Wenjie Zhong;Yusuke Miyao
中科院分区：
文献类型：
--
作者：
Wenjie Zhong;Yusuke Miyao
近年来，控制图像字幕的生成引起了人们的广泛关注。在本文中，我们提出了一个框架，利用部分句法依赖树作为控制信号，使图像字幕包括指定的单词和它们的句法结构。为了实现这一目的，我们提出了一个语法依赖结构感知模型（SDSAM），明确学习生成的图像字幕的语法结构，包括给定的部分依赖树。此外，我们提出了一个指标，以评估有多少指定的单词和它们的句法依赖关系包括在生成的字幕。我们在两个标准数据集上进行了实验：Microsoft COCO和Flickr 30 k。实验结果表明，我们的模型生成的图像字幕有效地控制在指定的单词和它们的句法结构。
Controlling the generation of image captions attracts lots of attention recently. In this paper, we propose a framework leveraging partial syntactic dependency trees as control signals to make image captions include specified words and their syntactic structures. To achieve this purpose, we propose a Syntactic Dependency Structure Aware Model (SDSAM), which explicitly learns to generate the syntactic structures of image captions to include given partial dependency trees. In addition, we come up with a metric to evaluate how many specified words and their syntactic dependencies are included in generated captions. We carry out experiments on two standard datasets: Microsoft COCO and Flickr30k. Empirical results show that image captions generated by our model are effectively controlled in terms of specified words and their syntactic structures.The code is available on GitHub.