NSF-BSF: RI: Small: Efficient Transformers via Formal and Empirical Analysis

构建一个最先进的自然语言处理（NLP）系统需要花费数百万美元，因为它需要在数十亿个文本文档上训练一个具有数千亿个参数（2021年是三年前的1000倍）的神经网络架构。 目前，一个单一的架构，“Transformer”，用于问答，摘要，机器翻译，分析和生成系统，文本分类，以及几乎所有其他的NLP研究系统。Transformer成本的显著降低将降低世界各地绝大多数研究小组参与研究的障碍，并减少NLP研究的环境足迹。降低成本的原则性方法也有望很容易地转移到几代模型中，这些模型将不可避免地取代Transformer。该项目从标准注意力函数的随机近似开始，该函数将Transformer的运行时间和内存要求从二次减少到线性（在输入长度上）。 对于这种随机化方法，应用了“理性模型”的透镜。理性模型为NLP（卷积和递归网络）中流行的前几代神经模型提供了统一的视图，并提高了计算效率和可解释性。 第二个研究方向集中在基于梯度的训练算法的效率。 经验证据表明，神经网络学习分为两个阶段：对超参数敏感的快速阶段和更鲁棒的慢速阶段。 该项目确定了该模式与当前NLP模型保持一致的程度，然后寻求利用该模式来加速第二阶段。 这两个方向都将使Transformer架构更高效，并显著降低其财务和环境成本，并可能为未来的神经网络架构做同样的事情。 该项目的实施将作为开放源代码软件提供，并提供友好的许可证，允许广泛采用。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Building a state-of-the-art natural language processing (NLP) system costs millions of dollars, because it requires training a neural network architecture with hundreds of billions of parameters (1000 times more in 2021 than just three years ago) on billions of text documents. At present, a single architecture, the "transformer", is used in question answering, summarization, machine translation, analysis and generation systems, text classification, and virtually every other NLP research system. A significant reduction in the transformer's costs will lower barriers to participation in research for the vast majority of research groups around the world and reduce the environmental footprint of NLP research. Principled methods for reducing that cost are also expected to transfer readily to the generations of models that will, inevitably, replace the transformer.This project begins with a randomized approximation to the standard attention function that reduces runtime and memory requirements of the transformer from quadratic to linear (in the input length). To this randomized approach, the lens of "rational models" is applied. Rational models have offered a unifying view of earlier generations of neural models popular in NLP (convolutional and recurrent networks) and gave rise to computational efficiency and interpretability gains. A second research direction focuses on the efficiency of gradient-based training algorithms. Empirical evidence has shown that neural network learning proceeds in two phases: a fast phase that is sensitive to hyperparameters and then a slow one that is more robust. This project establishes the extent to which the pattern holds with current NLP models and then seeks to exploit the pattern to speed up the second stage. Both directions will make the transformer architecture more efficient and significantly reduce its financial and environmental costs, and potentially do the same for future neural network architectures. The project's implementations will be made available as open-source software with friendly licenses permitting wide adoption.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.