FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attention
FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attention
复制标题
DOI：
--
复制
发表时间：
2021-08
期刊：
影响因子：
--
通讯作者：
T. Nguyen;Vai Suliafu;S. Osher;Long Chen;Bao Wang
中科院分区：
文献类型：
--
作者：
T. Nguyen;Vai Suliafu;S. Osher;Long Chen;Bao Wang
我们提出了 FMMformers，这是一类高效灵活的变压器，其灵感来自于著名的快速多极子方法（FMM），用于加速相互作用粒子模拟。 FMM 将粒子间相互作用分解为近场和远场分量，然后分别执行直接和粗粒度计算。类似地，FMMformers 将注意力分解为近场注意力和远场注意力，通过带状矩阵对近场注意力进行建模，通过低秩矩阵对远场注意力进行建模。计算 FMMformers 的注意力矩阵需要计算时间和内存占用相对于序列长度的线性复杂性。相比之下，标准变压器的复杂性却是二次方的。我们在 Long Range Arena 和语言建模基准上分析并验证了 FMMformers 相对于标准 Transformer 的优势。 FMMformers 在精度方面甚至可以大幅超越标准变压器。例如，FMMformers 在五个 Long Range Arena 任务中实现了 $60.74\%$ 的平均分类准确度，这明显优于标准 Transformer 的平均分类准确度 $58.70\%$。
We propose FMMformers, a class of efficient and flexible transformers inspired by the celebrated fast multipole method (FMM) for accelerating interacting particle simulation. FMM decomposes particle-particle interaction into near-field and far-field components and then performs direct and coarse-grained computation, respectively. Similarly, FMMformers decompose the attention into near-field and far-field attention, modeling the near-field attention by a banded matrix and the far-field attention by a low-rank matrix. Computing the attention matrix for FMMformers requires linear complexity in computational time and memory footprint with respect to the sequence length. In contrast, standard transformers suffer from quadratic complexity. We analyze and validate the advantage of FMMformers over the standard transformer on the Long Range Arena and language modeling benchmarks. FMMformers can even outperform the standard transformer in terms of accuracy by a significant margin. For instance, FMMformers achieve an average classification accuracy of $60.74\%$ over the five Long Range Arena tasks, which is significantly better than the standard transformer's average accuracy of $58.70\%$.