NSF-BSF: RI: Small: Efficient Transformers via Formal and Empirical Analysis

建立一个最先进的自然语言处理(NLP)系统耗资数百万美元，因为它需要训练一个具有数千亿个参数的神经网络结构(2021年是三年前的1000倍)。目前，在问答、摘要、机器翻译、分析和生成系统、文本分类以及几乎所有其他自然语言处理研究系统中都使用了单一的架构，即“转换器”。变压器成本的显著降低将降低世界各地绝大多数研究小组参与研究的门槛，并减少NLP研究的环境足迹。降低这一成本的原则性方法也有望很容易地转移到将不可避免地取代变压器的模型的世代。该项目开始于对标准注意函数的随机近似，该函数将变压器的运行时间和内存需求从二次减少到线性(在输入长度上)。对于这种随机化的方法，应用了“理性模型”的镜头。有理模型提供了在NLP(卷积和递归网络)中流行的前几代神经模型的统一视图，并提高了计算效率和可解释性。第二个研究方向是基于梯度的训练算法的效率。经验证据表明，神经网络学习分两个阶段进行：对超参数敏感的快速阶段和更稳健的缓慢阶段。该项目建立了该模式在当前NLP模型中的适用范围，然后寻求利用该模式来加速第二阶段。这两个方向都将使变压器体系结构更高效，并显著降低其财务和环境成本，并可能对未来的神经网络体系结构起到同样的作用。该项目的实施将以开放源码软件的形式提供，并拥有允许广泛采用的友好许可证。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Building a state-of-the-art natural language processing (NLP) system costs millions of dollars, because it requires training a neural network architecture with hundreds of billions of parameters (1000 times more in 2021 than just three years ago) on billions of text documents. At present, a single architecture, the "transformer", is used in question answering, summarization, machine translation, analysis and generation systems, text classification, and virtually every other NLP research system. A significant reduction in the transformer's costs will lower barriers to participation in research for the vast majority of research groups around the world and reduce the environmental footprint of NLP research. Principled methods for reducing that cost are also expected to transfer readily to the generations of models that will, inevitably, replace the transformer.This project begins with a randomized approximation to the standard attention function that reduces runtime and memory requirements of the transformer from quadratic to linear (in the input length). To this randomized approach, the lens of "rational models" is applied. Rational models have offered a unifying view of earlier generations of neural models popular in NLP (convolutional and recurrent networks) and gave rise to computational efficiency and interpretability gains. A second research direction focuses on the efficiency of gradient-based training algorithms. Empirical evidence has shown that neural network learning proceeds in two phases: a fast phase that is sensitive to hyperparameters and then a slow one that is more robust. This project establishes the extent to which the pattern holds with current NLP models and then seeks to exploit the pattern to speed up the second stage. Both directions will make the transformer architecture more efficient and significantly reduce its financial and environmental costs, and potentially do the same for future neural network architectures. The project's implementations will be made available as open-source software with friendly licenses permitting wide adoption.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.