Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps-猫眼课题宝

DOI：

发表时间：

2020-04

期刊：

ArXiv

影响因子：

通讯作者：

中科院分区：

其他

文献类型：

作者：

Tri Dao;N. Sohoni;Albert Gu;Matthew Eichhorn;Amit Blonder;Megan Leszczynski;A. Rudra;Christopher Ré

现代神经网络架构使用结构化的线性变换，如低秩矩阵、稀疏矩阵、置换和傅立叶变换，与一般线性映射相比，可以提高推理速度并减少内存使用。然而，选择使用无数结构化转换中的哪一个（及其相关的参数化）是一项费力的任务，需要权衡速度、空间和准确性。我们考虑一种不同的方法：我们引入了一个家庭的矩阵称为万花筒矩阵（K矩阵），可证明捕获任何结构矩阵与接近最佳的空间（参数）和时间（算术运算）的复杂性。我们通过经验验证，K矩阵可以在端到端管道中自动学习，以取代手工制作的过程，从而提高模型质量。例如，在ShuffleNet中替换通道洗牌可以将ImageNet的分类准确率提高5%。可学习的K矩阵还可以简化手工设计的管道--我们用万花筒层取代语音数据预处理中的滤波器组特征计算，仅导致TIMIT语音识别任务的准确性损失0.4%。K矩阵还可以捕获模型中的潜在结构：对于具有挑战性的置换图像分类任务，将K矩阵添加到标准卷积架构中可以学习潜在置换并将准确性提高8个点以上。我们提供了一个实际有效的实现我们的方法，并在Transformer网络中使用K-矩阵，以获得36%的语言翻译任务的端到端的推理速度快。

Modern neural network architectures use structured linear transformations, such as low-rank matrices, sparse matrices, permutations, and the Fourier transform, to improve inference speed and reduce memory usage compared to general linear maps. However, choosing which of the myriad structured transformations to use (and its associated parameterization) is a laborious task that requires trading off speed, space, and accuracy. We consider a different approach: we introduce a family of matrices called kaleidoscope matrices (K-matrices) that provably capture any structured matrix with near-optimal space (parameter) and time (arithmetic operation) complexity. We empirically validate that K-matrices can be automatically learned within end-to-end pipelines to replace hand-crafted procedures, in order to improve model quality. For example, replacing channel shuffles in ShuffleNet improves classification accuracy on ImageNet by up to 5%. Learnable K-matrices can also simplify hand-engineered pipelines---we replace filter bank feature computation in speech data preprocessing with a kaleidoscope layer, resulting in only 0.4% loss in accuracy on the TIMIT speech recognition task. K-matrices can also capture latent structure in models: for a challenging permuted image classification task, adding a K-matrix to a standard convolutional architecture can enable learning the latent permutation and improve accuracy by over 8 points. We provide a practically efficient implementation of our approach, and use K-matrices in a Transformer network to attain 36% faster end-to-end inference speed on a language translation task.

Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps

文献摘要

相似文献