Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
复制标题
DOI：
10.18653/v1/2022.emnlp-main.340
复制
发表时间：
2022-04
期刊：
影响因子：
--
通讯作者：
Yizhong Wang;Swaroop Mishra;Pegah Alipoormolabashi;Yeganeh Kordi;Amirreza Mirzaei;Anjana Arunkumar;Arjun Ashok;Arut Selvan Dhanasekaran;Atharva Naik;David Stap;Eshaan Pathak;Giannis Karamanolakis;H. Lai;I. Purohit;Ishani Mondal;Jacob Anderson;Kirby Kuznia;Krima Doshi;Maitreya Patel;Kuntal Kumar Pal;M. Moradshahi;Mihir Parmar;Mirali Purohit;Neeraj Varshney;Phani Rohitha Kaza;Pulkit Verma;Ravsehaj Singh Puri;Rushang Karia;Shailaja Keyur Sampat;Savan Doshi;Siddhartha Mishra;Sujan Reddy;Sumanta Patro;Tanay Dixit;Xudong Shen;Chitta Baral;Yejin Choi;Noah A. Smith;Hannaneh Hajishirzi;Daniel Khashabi
中科院分区：
文献类型：
--
作者：
Yizhong Wang;Swaroop Mishra;Pegah Alipoormolabashi;Yeganeh Kordi;Amirreza Mirzaei;Anjana Arunkumar;Arjun Ashok;Arut Selvan Dhanasekaran;Atharva Naik;David Stap;Eshaan Pathak;Giannis Karamanolakis;H. Lai;I. Purohit;Ishani Mondal;Jacob Anderson;Kirby Kuznia;Krima Doshi;Maitreya Patel;Kuntal Kumar Pal;M. Moradshahi;Mihir Parmar;Mirali Purohit;Neeraj Varshney;Phani Rohitha Kaza;Pulkit Verma;Ravsehaj Singh Puri;Rushang Karia;Shailaja Keyur Sampat;Savan Doshi;Siddhartha Mishra;Sujan Reddy;Sumanta Patro;Tanay Dixit;Xudong Shen;Chitta Baral;Yejin Choi;Noah A. Smith;Hannaneh Hajishirzi;Daniel Khashabi
当提供任务指令时，NLP模型能在多大程度上推广到各种看不见的任务？为了解决这个问题，我们首先介绍了Super-NaturalInstructions，这是一个包含1，616个不同NLP任务及其专家编写指令的基准。我们的集合涵盖了76种不同的任务类型，包括但不限于分类，提取，填充，序列标记，文本重写和文本合成。这个庞大而多样化的任务集合使得在预防训练模型下严格的跨任务泛化基准测试能够遵循任务子集上的指令并在剩余的看不见的任务上对其进行评估。此外，我们构建了Tk-Instruct，一个经过训练的Transformer模型，可以遵循各种上下文指令（普通语言任务定义或k-shot示例）。我们的实验表明，Tk-Instruct优于现有的跟踪模型，如InstructGPT超过9%，尽管是一个数量级较小的基准。我们进一步分析泛化作为各种缩放参数的函数，例如观察到的任务的数量，每个任务的实例数量和模型大小。我们希望我们的数据集和模型能够促进未来朝着更通用的NLP模型发展。
How well can NLP models generalize to a variety of unseen tasks when provided with task instructions? To address this question, we first introduce Super-NaturalInstructions, a benchmark of 1,616 diverse NLP tasks and their expert-written instructions. Our collection covers 76 distinct task types, including but not limited to classification, extraction, infilling, sequence tagging, text rewriting, and text composition. This large and diverse collection of tasks enables rigorous benchmarking of cross-task generalization under instructions—training models to follow instructions on a subset of tasks and evaluating them on the remaining unseen ones.Furthermore, we build Tk-Instruct, a transformer model trained to follow a variety of in-context instructions (plain language task definitions or k-shot examples). Our experiments show that Tk-Instruct outperforms existing instruction-following models such as InstructGPT by over 9% on our benchmark despite being an order of magnitude smaller. We further analyze generalization as a function of various scaling parameters, such as the number of observed tasks, the number of instances per task, and model sizes. We hope our dataset and model facilitate future progress towards more general-purpose NLP models.