Efficient NLP Model Finetuning via Multistage Data Filtering
Efficient NLP Model Finetuning via Multistage Data Filtering
复制标题
DOI：
10.24963/ijcai.2023/455
复制
发表时间：
2022-07
期刊：
影响因子：
--
通讯作者：
Ouyang Xu;S. Ansari;F. Lin;Yangfeng Ji
中科院分区：
文献类型：
--
作者：
Ouyang Xu;S. Ansari;F. Lin;Yangfeng Ji
由于模型填充是现代NLP的核心，我们将其效率最大化。以流式训练方式与目标模型相关的训练示例一个进一步的前向训练的元预测因素。高达6.8×，而仅看到较小的精度降解也是有效的，即使训练一个训练示例仅遇到一次。现有的登录技术可在以下网址提供：
As model finetuning is central to the modern NLP, we set to maximize its efficiency. Motivated by redundancy in training examples and the sheer sizes of pretrained models, we exploit a key opportunity: training only on important data. To this end, we set to filter training examples in a streaming fashion, in tandem with training the target model. Our key techniques are two: (1) automatically determine a training loss threshold for skipping backward training passes; (2) run a meta predictor for further skipping forward training passes. We integrate the above techniques in a holistic, three-stage training pro- cess. On a diverse set of benchmarks, our method reduces the required training examples by up to 5.3× and training time by up to 6.8×, while only seeing minor accuracy degradation. Our method is effective even for training one epoch, where each training example is encountered only once. It is simple to implement and is compatible with the existing finetuning techniques. Code is available at: https://github.com/xo28/efficient-NLP-multistage-training