Persistent RNNs: Stashing Weights on Chip
Persistent RNNs: Stashing Weights on Chip
复制标题
持久 RNN：将权重存储在芯片上
DOI：
--
复制
发表时间：
2016
期刊：
影响因子：
--
通讯作者：
S. Satheesh
中科院分区：
文献类型：
--
作者：
G. Diamos;Shubho Sengupta;Bryan Catanzaro;Mike Chrzanowski;Adam Coates;Erich Elsen;Jesse Engel;Awni Y. Hannun;S. Satheesh
本文介绍了一种将递归神经网络（RNN）架构有效映射到并行处理器（如GPU）的框架。我们的方法的关键是使用持久的计算内核，利用处理器的内存层次结构，在多个时间步重用网络权重。使用我们的框架，我们展示了如何在较低的小批量大小下实现比基于矩阵乘法的RNN直接实现更高的计算吞吐量。我们的初始实现在NVIDIA TitanX GPU上以4的小批量大小实现了2.8 TFLOP/s，这大约是理论峰值吞吐量的45%，并且比基于优化的GEMM内核的标准RNN实现快30倍。将每个处理器的批处理大小从64减少到4，可以将激活内存占用减少16倍，使用数据并行性可以将GPU扩展到16倍，并允许我们有效地探索具有多达108个剩余RNN层的端到端语音识别模型。
This paper introduces a framework for mapping Recurrent Neural Network (RNN) architectures efficiently onto parallel processors such as GPUs. Key to our approach is the use of persistent computational kernels that exploit the processor’s memory hierarchy to reuse network weights over multiple timesteps. Using our framework, we show how it is possible to achieve substantially higher computational throughput at lower mini-batch sizes than direct implementations of RNNs based on matrix multiplications. Our initial implementation achieves 2.8 TFLOP/s at a mini-batch size of 4 on an NVIDIA TitanX GPU, which is about 45% of theoretical peak throughput, and is 30X faster than a standard RNN implementation based on optimized GEMM kernels at this batch size. Reducing the batch size from 64 to 4 per processor provides a 16x reduction in activation memory footprint, enables strong scaling to 16x more GPUs using data-parallelism, and allows us to efficiently explore end-to-end speech recognition models with up to 108 residual RNN layers.
DOI：
--
发表时间：
2015-12
期刊：
--
影响因子：
--
作者：
Dario Amodei;S. Ananthanarayanan;Rishita Anubhai;Jin Bai;Eric Battenberg;Carl Case;J. Casper;
通讯作者：
Dario Amodei;S. Ananthanarayanan;Rishita Anubhai;Jin Bai;Eric Battenberg;Carl Case;J. Casper;