Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms
Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms
复制标题
DOI：
--
复制
发表时间：
2018-02
期刊：
影响因子：
--
通讯作者：
Ashok Vardhan Makkuva;P. Viswanath;Sreeram Kannan;Sewoong Oh
中科院分区：
文献类型：
--
作者：
Ashok Vardhan Makkuva;P. Viswanath;Sreeram Kannan;Sewoong Oh
混合专家（莫伊）是一种广泛流行的集成学习模型，是非常成功的现代神经网络的基本构建块，也是门控递归单元（GRU）和注意力网络的一个组成部分。然而，目前用于学习莫伊的算法，包括EM算法和梯度下降已知陷入局部最优。从理论的角度来看，寻找一个有效的和可证明的一致的算法来学习参数仍然是一个长期存在的开放问题超过二十年。在本文中，我们介绍了第一个算法，学习真正的参数的莫伊模型的广泛的非线性与全球一致性的保证。现有的算法联合或迭代估计专家参数和门控参数的莫伊，我们提出了一种新的算法，打破了僵局，可以直接估计专家参数，通过感知其回声在一个精心设计的输入和输出之间的交叉矩张量。一旦专家是已知的，门控参数的恢复仍然需要一个EM算法，但是，我们表明，这个简化的问题的EM算法，不像联合EM算法，收敛到真正的参数。我们经验验证我们的算法的合成和真实的数据集在各种设置，并显示上级性能标准基线。
Mixture-of-Experts (MoE) is a widely popular model for ensemble learning and is a basic building block of highly successful modern neural networks as well as a component in Gated Recurrent Units (GRU) and Attention networks. However, present algorithms for learning MoE including the EM algorithm, and gradient descent are known to get stuck in local optima. From a theoretical viewpoint, finding an efficient and provably consistent algorithm to learn the parameters remains a long standing open problem for more than two decades. In this paper, we introduce the first algorithm that learns the true parameters of a MoE model for a wide class of non-linearities with global consistency guarantees. While existing algorithms jointly or iteratively estimate the expert parameters and the gating paramters in the MoE, we propose a novel algorithm that breaks the deadlock and can directly estimate the expert parameters by sensing its echo in a carefully designed cross-moment tensor between the inputs and the output. Once the experts are known, the recovery of gating parameters still requires an EM algorithm; however, we show that the EM algorithm for this simplified problem, unlike the joint EM algorithm, converges to the true parameters. We empirically validate our algorithm on both the synthetic and real data sets in a variety of settings, and show superior performance to standard baselines.