Learning Task-Agnostic Embedding of Multiple Black-Box Experts for Multi-Task Model Fusion
Learning Task-Agnostic Embedding of Multiple Black-Box Experts for Multi-Task Model Fusion
复制标题
DOI：
--
复制
发表时间：
2020-07
期刊：
影响因子：
--
通讯作者：
T. Hoang;Chi Thanh Lam;B. Low;Patrick Jaillet
中科院分区：
文献类型：
--
作者：
T. Hoang;Chi Thanh Lam;B. Low;Patrick Jaillet
模型融合是集体学习中的一项新兴研究，其中具有私有数据和学习架构的异构专家需要结合他们的黑盒知识以获得更好的性能。现有文献通过局部知识蒸馏方案来实现这一目标，该方案将每个预先训练的专家的预测模式传输到白盒模仿器模型上，该模型可以有效地合并到全局模型中。然而，该方案并没有扩展到多任务场景，在多任务场景中，不同的专家接受过培训来解决不同的任务，并且他们仅部分提炼的知识与新任务相关。为了解决这个多任务挑战，我们开发了一种新的融合范式，将每个专家表示为一系列预测原型的分布，这些预测原型与原型分布中编码的特定于任务的信息隔离。然后可以重新集成与任务无关的原型以生成新模型，该模型解决使用不同原型分布编码的新任务。所提出的框架的融合和适应性能在几个现实世界的基准数据集上得到了实证证明。
Model fusion is an emerging study in collective learning where heterogeneous experts with private data and learning architectures need to combine their black-box knowledge for better performance. Existing literature achieves this via a local knowledge distillation scheme that transfuses the predictive patterns of each pre-trained expert onto a white-box imitator model, which can be incorporated efficiently into a global model. This scheme however does not extend to multi-task scenarios where different experts were trained to solve different tasks and only part of their distilled knowledge is relevant to a new task. To address this multi-task challenge, we develop a new fusion paradigm that represents each expert as a distribution over a spectrum of predictive prototypes, which are isolated from task-specific information encoded within the prototype distribution. The task-agnostic prototypes can then be reintegrated to generate a new model that solves a new task encoded with a different prototype distribution. The fusion and adaptation performance of the proposed framework is demonstrated empirically on several real-world benchmark datasets.