Deep Mixture of Experts via Shallow Embedding-猫眼课题宝

DOI：

发表时间：

2018-06

期刊：

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Xin Wang;F. Yu;Lisa Dunlap;Yian Ma;Yi-An Ma;Ruth Wang;Azalia Mirhoseini;Trevor Darrell;Joseph Gonzalez

更大的网络通常具有更大的代表能力，但代价是增加了计算复杂性。稀疏化这种网络一直是一个活跃的研究领域，但通常限于静态正则化或使用强化学习的动态方法。我们探索了一种混合专家（莫伊）方法来进行深度动态路由，该方法在每个示例的基础上激活网络中的某些专家。我们的新型DeepMoE架构通过自适应稀疏化和重新校准每个卷积层中的通道特征来提高标准卷积网络的表示能力。我们采用多头稀疏门控网络来确定每个输入的通道的选择和缩放，利用单个卷积网络中专家的指数组合。我们提出的架构在四个基准数据集和任务上进行了评估，我们表明Deep-MoE能够以比标准卷积网络更低的计算量实现更高的准确性。

Larger networks generally have greater representational power at the cost of increased computational complexity. Sparsifying such networks has been an active area of research but has been generally limited to static regularization or dynamic approaches using reinforcement learning. We explore a mixture of experts (MoE) approach to deep dynamic routing, which activates certain experts in the network on a per-example basis. Our novel DeepMoE architecture increases the representational power of standard convolutional networks by adaptively sparsifying and recalibrating channel-wise features in each convolutional layer. We employ a multi-headed sparse gating network to determine the selection and scaling of channels for each input, leveraging exponential combinations of experts within a single convolutional network. Our proposed architecture is evaluated on four benchmark datasets and tasks, and we show that Deep-MoEs are able to achieve higher accuracy with lower computation than standard convolutional networks.

Deep Mixture of Experts via Shallow Embedding

文献摘要

相似文献