Bayesian tensorized neural networks with automatic rank selection
Bayesian tensorized neural networks with automatic rank selection
复制标题
DOI：
10.1016/j.neucom.2021.04.117
复制
发表时间：
2021-05-19
期刊：
影响因子：
6
通讯作者：
Zhang, Zheng
中科院分区：
文献类型：
--
作者：
Hawkins, Cole;Zhang, Zheng
张量分解是压缩过参数神经网络并使其能够在资源受限的硬件平台上部署的一种有效方法。然而，在训练过程中直接应用张量压缩是一项具有挑战性的任务，因为很难选择合适的张量等级。为了解决这一挑战，本文提出了一种低阶贝叶斯张化神经网络。我们的贝叶斯方法通过自适应张量等级确定来执行自动模型压缩。我们还提出了后验密度计算和最大后验概率(MAP)估计的方法，用于端到端的张化神经网络的训练。我们在一个两层完全连接的神经网络、一个6层的CNN和一个110层的残差神经网络上进行了实验验证，我们的工作直接从训练中产生了7.4x到137x的紧凑神经网络，同时获得了高的预测精度。(C)2021年爱思唯尔B.V.保留所有权利。
Tensor decomposition is an effective approach to compress over-parameterized neural networks and to enable their deployment on resource-constrained hardware platforms. However, directly applying tensor compression in the training process is a challenging task due to the difficulty of choosing a proper tensor rank. In order to address this challenge, this paper proposes a low-rank Bayesian tensorized neural network. Our Bayesian method performs automatic model compression via an adaptive tensor rank determination. We also present approaches for posterior density calculation and maximum a posteriori (MAP) estimation for the end-to-end training of our tensorized neural network. We provide experimental validation on a two-layer fully connected neural network, a 6-layer CNN and a 110-layer residual neural network where our work produces 7.4x to 137x more compact neural networks directly from the training while achieving high prediction accuracy. (C) 2021 Elsevier B.V. All rights reserved.