CCF: SHF: Small: Transformer synthesis

在首次提出的短短四年内，Transformer 就对自然语言处理 (NLP) 领域产生了巨大影响，并且也开始对计算机视觉等其他领域产生影响。这一成功很大程度上得益于大规模预训练数据集、不断增强的计算能力和强大的训练技术。然而，仍然存在的一个主要挑战是针对特定任务和一组用户需求进行高效的最佳变压器模型综合。然而，这并不容易做到，因为变压器模型的设计空间很大。 该项目通过开发变压器综合方法和工具来应对这一挑战。 鉴于变压器的重要性，此类工具可能会对许多应用领域产生变革性影响。该研究将通过技术转让向业界传播，例如通过源代码的开源在线分发、暑期实习以及利用 PI 与当地公司的合作。外展和课程开发计划也将在拟议研究的背景下进行。目前还没有可以驾驭广阔的变压器超参数设计空间的通用框架。先前提出的变压器模型在通过网络的数据流方面是同质的。不幸的是，这会导致变压器架构非常次优。该项目扩展了变压器的设计空间，纳入了异构架构，通过采用卷积和线性变换等其他操作来超越自注意力。它还将探索新颖的投影层和位置编码，以使隐藏尺寸在各种变换器层上灵活。它将使用密集嵌入来捕获模型相似性，以显着提高搜索效率。它将开发异方差代理模型以进一步加快搜索速度。它将包括优化长输入序列的远程交互的操作。它还将探索跳过连接和块级生长和修剪综合，以提高架构搜索效率。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力价值和更广泛的影响审查标准进行评估，被认为值得支持。

Just within four years of being first proposed, transformers have had a dramatic impact on the natural language processing (NLP) field and are also beginning to have an impact on other fields, such as computer vision. This success has largely been driven by large-scale pre-training datasets, increasing computational power, and robust training techniques. However, a major challenge that remains is efficient optimal transformer model synthesis for a specific task and set of user requirements. However, this is not easy to do since the design space of transformer models is vast. This project addresses this challenge through the development of transformer-synthesis methodologies and tools. Given the importance of transformers, such tools are likely to have a transformative impact on many application areas. The research will be disseminated to industry via tech transfer e.g., via open-source online distribution of source codes, summer internships, and by leveraging PIs involvement with local companies. Outreach and curriculum development plans will also be undertaken within the context of the proposed research.There is currently no universal framework that can navigate the vast transformer hyperparameter design space. Previously proposed transformer models are homogeneous in terms of data flow through the network. Unfortunately, this leads to very suboptimal transformer architectures. This project expands the transformer design space to incorporate heterogeneous architectures that venture beyond self-attention by employing other operations like convolutions and linear transforms. It will also explore novel projection layers and positional encodings to make hidden sizes flexible across various transformer layers. It will use a dense embedding to capture model similarity to significantly enhance search efficiency. It will develop a heteroscedastic surrogate model to further speed up search. It will include operations that optimize long-range interactions for long input sequences. It will also explore skipped connections and block-level grow-and-prune synthesis to improve architectural search efficiency.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.