MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning
MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning
复制标题
DOI：
10.48550/arxiv.2212.10773
复制
发表时间：
2022-12
期刊：
影响因子：
--
通讯作者：
Zhiyang Xu;Ying Shen;Lifu Huang
中科院分区：
文献类型：
--
作者：
Zhiyang Xu;Ying Shen;Lifu Huang
指令调整是一种新的学习范式，它对通过指令指定的任务微调预先训练的语言模型，在各种自然语言处理任务上显示出良好的零命中率性能。然而，它还没有被探索用于视觉和多模式任务。在这项工作中，我们介绍了MultiInstruct，这是第一个多通道指令优化基准数据集，由62个不同的多通道任务组成，以统一的SEQ-to-SEQ格式覆盖10个大类。这些任务来自现有的21个开源数据集，每个任务配备了5个专家编写的说明。我们以OFA作为多通道指令调优的基础预训练模型，为了进一步提高其零命中性能，我们探索了多种迁移学习策略来利用大规模的自然指令数据集。实验结果表明，在各种不可见的多模式任务上具有很强的零射性能，并且从纯文本的教学数据集中进行迁移学习具有很好的效果。我们还设计了一种新的评估指标-敏感度，来评估模型对各种指令的敏感度。我们的结果表明，在不同的任务和指令集上微调模型会降低对每个任务的指令变化的敏感度。
Instruction tuning, a new learning paradigm that fine-tunes pre-trained language models on tasks specified through instructions, has shown promising zero-shot performance on various natural language processing tasks. However, it has yet to be explored for vision and multimodal tasks. In this work, we introduce MultiInstruct, the first multimodal instruction tuning benchmark dataset that consists of 62 diverse multimodal tasks in a unified seq-to-seq format covering 10 broad categories. The tasks are derived from 21 existing open-source datasets and each task is equipped with 5 expert-written instructions. We take OFA as the base pre-trained model for multimodal instruction tuning, and to further improve its zero-shot performance, we explore multiple transfer learning strategies to leverage the large-scale Natural Instructions dataset. Experimental results demonstrate strong zero-shot performance on various unseen multimodal tasks and the benefit of transfer learning from a text-only instruction dataset. We also design a new evaluation metric – Sensitivity, to evaluate how sensitive the model is to the variety of instructions. Our results indicate that fine-tuning the model on a diverse set of tasks and instructions leads to a reduced sensitivity to variations in instructions for each task.