CCF: SHF: Small: Transformer synthesis

在最初提出的四年内，转换器已经对自然语言处理(NLP)领域产生了巨大的影响，并开始对其他领域产生影响，例如计算机视觉。这一成功在很大程度上是由大规模的预训练数据集、不断增加的计算能力和稳健的训练技术推动的。然而，仍然存在的一个主要挑战是针对特定任务和一组用户需求的高效最优变压器模型综合。然而，要做到这一点并不容易，因为变压器模型的设计空间广阔。该项目通过开发变压器综合方法和工具来应对这一挑战。鉴于变压器的重要性，这类工具可能会对许多应用领域产生变革性的影响。这项研究将通过技术转让向业界传播，例如通过开源在线分发源代码、暑期实习以及利用私人投资机构与当地公司的参与。外展和课程开发计划也将在拟议研究的范围内进行。目前还没有一个通用框架可以驾驭庞大的变压器超参数设计空间。以前提出的变压器模型在通过网络的数据流方面是同类的。不幸的是，这导致了非常不理想的变压器架构。该项目扩展了变压器的设计空间，通过使用卷积和线性变换等其他操作，整合了超越自我关注的异类体系结构。它还将探索新的投影层和位置编码，使隐藏的尺寸在不同的变压器层之间灵活。它将使用密集嵌入来捕获模型相似度，从而显著提高搜索效率。它将开发一个异方差代理模型，以进一步加快搜索速度。它将包括优化长输入序列的远程交互的操作。它还将探索跳过的连接和块级增长和修剪综合，以提高架构搜索效率。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Just within four years of being first proposed, transformers have had a dramatic impact on the natural language processing (NLP) field and are also beginning to have an impact on other fields, such as computer vision. This success has largely been driven by large-scale pre-training datasets, increasing computational power, and robust training techniques. However, a major challenge that remains is efficient optimal transformer model synthesis for a specific task and set of user requirements. However, this is not easy to do since the design space of transformer models is vast. This project addresses this challenge through the development of transformer-synthesis methodologies and tools. Given the importance of transformers, such tools are likely to have a transformative impact on many application areas. The research will be disseminated to industry via tech transfer e.g., via open-source online distribution of source codes, summer internships, and by leveraging PIs involvement with local companies. Outreach and curriculum development plans will also be undertaken within the context of the proposed research.There is currently no universal framework that can navigate the vast transformer hyperparameter design space. Previously proposed transformer models are homogeneous in terms of data flow through the network. Unfortunately, this leads to very suboptimal transformer architectures. This project expands the transformer design space to incorporate heterogeneous architectures that venture beyond self-attention by employing other operations like convolutions and linear transforms. It will also explore novel projection layers and positional encodings to make hidden sizes flexible across various transformer layers. It will use a dense embedding to capture model similarity to significantly enhance search efficiency. It will develop a heteroscedastic surrogate model to further speed up search. It will include operations that optimize long-range interactions for long input sequences. It will also explore skipped connections and block-level grow-and-prune synthesis to improve architectural search efficiency.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.