Multi-task Multimodal Learning for Disaster Situation Assessment
Multi-task Multimodal Learning for Disaster Situation Assessment
复制标题
DOI：
10.1109/mipr49039.2020.00050
复制
发表时间：
2020-08
期刊：
影响因子：
--
通讯作者：
Tianyi Wang;Yudong Tao;Shu‐Ching Chen;Mei-Ling Shyu
中科院分区：
文献类型：
--
作者：
Tianyi Wang;Yudong Tao;Shu‐Ching Chen;Mei-Ling Shyu
在灾害发生时，应急小组需要在尽可能早的阶段制定应急计划。社交媒体平台包含丰富的信息，有助于评估当前的形势。本文提出了一种新的具有自动损失加权的多任务多模态深度学习框架。我们的框架能够捕捉不同概念和数据模式之间的相关性。提出的自动损失加权方法可以避免繁琐的人工权重调整过程，提高模型的性能。在Twitter的大规模多模态灾害数据集上进行了广泛的实验，以确定灾后人道主义类别和基础设施损坏程度。结果表明，通过学习多个任务的共享潜在空间与损失加权，我们的模型可以优于所有的单任务。
During disaster events, emergency response teams need to draw up the response plan at the earliest possible stage. Social media platforms contain rich information which could help to assess the current situation. In this paper, a novel multi-task multimodal deep learning framework with automatic loss weighting is proposed. Our framework is able to capture the correlation among different concepts and data modalities. The proposed automatic loss weighting method can prevent the tedious manual weight tuning process and improve the model performance. Extensive experiments on a large-scale multimodal disaster dataset from Twitter are conducted to identify post-disaster humanitarian category and infrastructure damage level. The results show that by learning the shared latent space of multiple tasks with loss weighting, our model can outperform all single tasks.