Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors
Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors
复制标题
DOI：
10.48550/arxiv.2205.10279
复制
发表时间：
2022-05
期刊：
影响因子：
--
通讯作者：
Ravid Shwartz-Ziv;Micah Goldblum;Hossein Souri;Sanyam Kapoor;Chen Zhu;Yann LeCun;A. Wilson
中科院分区：
文献类型：
--
作者：
Ravid Shwartz-Ziv;Micah Goldblum;Hossein Souri;Sanyam Kapoor;Chen Zhu;Yann LeCun;A. Wilson
深度学习正越来越多地朝着迁移学习范式发展，即从源任务上学习的初始化开始，在下游任务上对大型基础模型进行微调。但是初始化包含的关于源任务的信息相对较少。相反，我们表明，我们可以从源任务中学习高度信息化的后验，通过监督或自我监督的方法，然后作为修改下游任务的整个损失表面的先验的基础。这种简单的模块化方法可以在各种下游分类和分割任务上实现显着的性能提升和更有效的数据学习，作为标准预训练策略的替代品。这些高度信息化的先验也可以保存以供将来使用，类似于预先训练的权重，并且与贝叶斯深度学习中通常使用的零均值各向同性无信息先验形成对比。
Deep learning is increasingly moving towards a transfer learning paradigm whereby large foundation models are fine-tuned on downstream tasks, starting from an initialization learned on the source task. But an initialization contains relatively little information about the source task. Instead, we show that we can learn highly informative posteriors from the source task, through supervised or self-supervised approaches, which then serve as the basis for priors that modify the whole loss surface on the downstream task. This simple modular approach enables significant performance gains and more data-efficient learning on a variety of downstream classification and segmentation tasks, serving as a drop-in replacement for standard pre-training strategies. These highly informative priors also can be saved for future use, similar to pre-trained weights, and stand in contrast to the zero-mean isotropic uninformative priors that are typically used in Bayesian deep learning.