Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Execution
Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Execution
复制标题
DOI：
10.1609/aaai.v32i1.11630
复制
发表时间：
2017-01
期刊：
影响因子：
--
通讯作者：
Lanlan Liu;Jia Deng
中科院分区：
文献类型：
--
作者：
Lanlan Liu;Jia Deng
我们介绍了动态深度神经网络（D2 NN），这是一种允许选择性执行的新型前馈深度神经网络。给定一个输入，只执行D2 NN神经元的一个子集，并且特定子集由D2 NN本身确定。通过根据输入修剪不必要的计算，D2 NN提供了一种提高计算效率的方法。为了实现动态选择性执行，D2 NN用控制器模块增强了前馈深度神经网络（可微模块的有向无环图）。每个控制器模块是一个子网络，其输出是控制其他模块是否可以执行的决策。D2 NN是端到端训练的。D2 NN中的常规模块和控制器模块都是可学习的，并经过联合训练以优化精度和效率。这种训练是通过将反向传播与强化学习相结合来实现的。通过对各种D2 NN架构在图像分类任务上的广泛实验，我们证明了D2 NN是通用和灵活的，并且可以有效地优化精度-效率权衡。
We introduce Dynamic Deep Neural Networks (D2NN), a new type of feed-forward deep neural network that allows selective execution. Given an input, only a subset of D2NN neurons are executed, and the particular subset is determined by the D2NN itself. By pruning unnecessary computation depending on input, D2NNs provide a way to improve computational efficiency. To achieve dynamic selective execution, a D2NN augments a feed-forward deep neural network (directed acyclic graph of differentiable modules) with controller modules. Each controller module is a sub-network whose output is a decision that controls whether other modules can execute. A D2NN is trained end to end. Both regular and controller modules in a D2NN are learnable and are jointly trained to optimize both accuracy and efficiency. Such training is achieved by integrating backpropagation with reinforcement learning. With extensive experiments of various D2NN architectures on image classification tasks, we demonstrate that D2NNs are general and flexible, and can effectively optimize accuracy-efficiency trade-offs.