ViNTER: Image Narrative Generation with Emotion-Arc-Aware Transformer
ViNTER: Image Narrative Generation with Emotion-Arc-Aware Transformer
复制标题
DOI：
10.1145/3487553.3524649
复制
发表时间：
2022-02
期刊：
影响因子：
--
通讯作者：
Kohei Uehara;Yusuke Mori;Yusuke Mukuta;Tatsuya Harada
中科院分区：
文献类型：
--
作者：
Kohei Uehara;Yusuke Mori;Yusuke Mukuta;Tatsuya Harada
图像叙事生成是从图像出发，以主观视角创造故事的任务。鉴于作家、读者和人物的主观感受在讲故事中的重要性，图像叙事的生成方法应该考虑人类的情感。在这项研究中，我们提出了一种新的图像叙事生成方法称为ViNTER（视觉叙事Transformer与情感弧表示），它以“情感弧”作为输入，以捕捉一系列的情感变化。由于情感弧代表了情感变化的轨迹，因此我们可以将故事中情感变化的详细信息包含到模型中。我们提出了自动和手动评估的图像叙事数据集的实验结果，并证明了所提出的方法的有效性。
Image narrative generation is a task to create a story from an image with a subjective viewpoint. Given the importance of the subjective feelings of writers, readers, and characters in storytelling, an image narrative generation method should consider human emotion. In this study, we propose a novel method of image narrative generation called ViNTER (Visual Narrative Transformer with Emotion arc Representation), which takes “emotion arc” as input to capture a sequence of emotional changes. Since emotion arcs represent the trajectory of emotional change, it is expected that we can include detailed information about the emotional changes in the story to the model. We present experimental results of both automatic and manual evaluations on the Image Narrative dataset and demonstrate the effectiveness of the proposed approach.