Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
复制标题
DOI：
--
复制
发表时间：
2020-10
期刊：
影响因子：
--
通讯作者：
Zirui Wang;Yulia Tsvetkov;Orhan Firat;Yuan Cao
中科院分区：
文献类型：
--
作者：
Zirui Wang;Yulia Tsvetkov;Orhan Firat;Yuan Cao
包含数十种甚至数百种语言的大规模多语言模型给多任务优化带来了巨大的挑战。虽然应用与语言无关的程序来优化联合多语言任务目标是一种常见的做法，但如何适当地描述和利用其潜在的问题结构来提高优化效率仍未得到充分探索。本文试图从损失函数几何的角度窥探多语言优化的黑匣子。我们发现，沿优化轨迹测量的梯度相似性是一个重要的信号，它不仅与语言贴近度相关，而且与整体模型的性能也有很好的相关性。这样的观察有助于我们识别现有基于梯度的多任务学习方法的关键限制，因此我们得出了一种简单且可扩展的优化过程，称为梯度疫苗，它鼓励在接近的任务中进行更多几何排列的参数更新。实验表明，我们的方法在多语言机器翻译和多语言模型Xtreme基准任务上获得了显著的模型性能提升。我们的工作揭示了适当测量和利用语言邻近度在多语言优化中的重要性，并对多语言建模以外的多任务学习具有更广泛的意义。
Massively multilingual models subsuming tens or even hundreds of languages pose great challenges to multi-task optimization. While it is a common practice to apply a language-agnostic procedure optimizing a joint multilingual task objective, how to properly characterize and take advantage of its underlying problem structure for improving optimization efficiency remains under-explored. In this paper, we attempt to peek into the black-box of multilingual optimization through the lens of loss function geometry. We find that gradient similarity measured along the optimization trajectory is an important signal, which correlates well with not only language proximity but also the overall model performance. Such observation helps us to identify a critical limitation of existing gradient-based multi-task learning methods, and thus we derive a simple and scalable optimization procedure, named Gradient Vaccine, which encourages more geometrically aligned parameter updates for close tasks. Empirically, our method obtains significant model performance gains on multilingual machine translation and XTREME benchmark tasks for multilingual language models. Our work reveals the importance of properly measuring and utilizing language proximity in multilingual optimization, and has broader implications for multi-task learning beyond multilingual modeling.