Model Fusion with Kullback-Leibler Divergence
Model Fusion with Kullback-Leibler Divergence
复制标题
DOI：
--
复制
发表时间：
2020-07
期刊：
影响因子：
--
通讯作者：
Sebastian Claici;M. Yurochkin;S. Ghosh;J. Solomon
中科院分区：
文献类型：
--
作者：
Sebastian Claici;M. Yurochkin;S. Ghosh;J. Solomon
我们提出了一种融合从异质数据集中学习的后验分布的方法。我们的算法依赖于融合模型和单个数据集后方的平均场假设，并使用简单的赋值平均方法进行处理。通过求解分配问题的正则化变体，将数据集后方的分量分配给所提出的全局模型分量。然后基于这些赋值，在KL发散下通过它们的平均值来更新全局分量。对于指数族变分分布，我们的公式导致了一个有效的非参数算法来计算融合模型。我们的算法易于描述和实现，在运动捕获分析、主题建模和贝叶斯神经网络的联邦学习方面具有较高的效率和竞争力。
We propose a method to fuse posterior distributions learned from heterogeneous datasets. Our algorithm relies on a mean field assumption for both the fused model and the individual dataset posteriors and proceeds using a simple assign-and-average approach. The components of the dataset posteriors are assigned to the proposed global model components by solving a regularized variant of the assignment problem. The global components are then updated based on these assignments by their mean under a KL divergence. For exponential family variational distributions, our formulation leads to an efficient non-parametric algorithm for computing the fused model. Our algorithm is easy to describe and implement, efficient, and competitive with state-of-the-art on motion capture analysis, topic modeling, and federated learning of Bayesian neural networks.