CCF: SHF: Small: Transformer synthesis

CCF：SHF：小型：变压器综合

• 批准号: 2203399
• 负责人: Niraj Jha
• 金额: $ 60万
• 依托单位: Princeton University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2022
• 资助国家: 美国
• 起止时间: 2022-02-01 至 2025-01-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2203399&HistoricalAwards=false
• 关键词: CCF; SHF; Small; Transformer; synthesis

Just within four years of being first proposed, transformers have had a dramatic impact on the natural language processing (NLP) field and are also beginning to have an impact on other fields, such as computer vision. This success has largely been driven by large-scale pre-training datasets, increasing computational power, and robust training techniques. However, a major challenge that remains is efficient optimal transformer model synthesis for a specific task and set of user requirements. However, this is not easy to do since the design space of transformer models is vast. This project addresses this challenge through the development of transformer-synthesis methodologies and tools. Given the importance of transformers, such tools are likely to have a transformative impact on many application areas. The research will be disseminated to industry via tech transfer e.g., via open-source online distribution of source codes, summer internships, and by leveraging PIs involvement with local companies. Outreach and curriculum development plans will also be undertaken within the context of the proposed research.There is currently no universal framework that can navigate the vast transformer hyperparameter design space. Previously proposed transformer models are homogeneous in terms of data flow through the network. Unfortunately, this leads to very suboptimal transformer architectures. This project expands the transformer design space to incorporate heterogeneous architectures that venture beyond self-attention by employing other operations like convolutions and linear transforms. It will also explore novel projection layers and positional encodings to make hidden sizes flexible across various transformer layers. It will use a dense embedding to capture model similarity to significantly enhance search efficiency. It will develop a heteroscedastic surrogate model to further speed up search. It will include operations that optimize long-range interactions for long input sequences. It will also explore skipped connections and block-level grow-and-prune synthesis to improve architectural search efficiency.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

在首次提出的四年内，transformers对自然语言处理（NLP）领域产生了巨大的影响，并且也开始对其他领域产生影响，例如计算机视觉。这一成功在很大程度上是由大规模的预训练数据集、不断提高的计算能力和强大的训练技术推动的。然而，一个主要的挑战，仍然是有效的最佳Transformer模型合成的特定任务和用户的要求。然而，这并不容易做到，因为Transformer模型的设计空间是巨大的。 该项目通过开发转化器合成方法和工具来应对这一挑战。 鉴于变压器的重要性，此类工具可能会对许多应用领域产生变革性的影响。该研究将通过技术转让传播给工业界，通过开源在线分发源代码，暑期实习，以及利用PI参与当地公司。外联和课程开发计划也将在拟议的研究范围内进行。目前还没有通用的框架，可以浏览广阔的Transformer超参数设计空间。以前提出的Transformer模型在通过网络的数据流方面是同质的。不幸的是，这导致非常次优的Transformer架构。该项目扩展了Transformer的设计空间，通过采用卷积和线性变换等其他操作，将异构体系结构纳入其中。它还将探索新的投影层和位置编码，使隐藏的大小灵活地跨各种Transformer层。它将使用密集嵌入来捕获模型相似性，以显着提高搜索效率。它将开发异方差代理模型，以进一步加快搜索速度。它将包括优化长输入序列的远程相互作用的操作。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。