SHF: Small: Improving Efficiency of Vision Transformers via Software-Hardware Co-Design and Acceleration

变形模型是机器学习领域的一个相对较新的突破，它彻底改变了自然语言处理，促进了计算机视觉模型的推广。然而，变压器模型的广泛采用需要使它们显著提高能源效率。Transformer模型过于复杂，现有的硬件对于它们的有效执行来说不是最佳的。在这个项目中，研究人员正在探索两个相互关联的研究问题来解决变压器的效率：1)创建新的变压器模型，可以动态修剪，以提高效率，而不牺牲精度；2)设计专门的硬件，使变压器的执行更有效。该项目的影响在几个方面是显著的：首先，它促进了研究界的科学进步，促进了社区对注意力作为变压器模型中的主要机制的理解，以及如何将其用于复杂学习模型的上下文感知修剪。其次，它扩展了硬件社区在设计复杂和可调硬件系统的动态精确调谐和调度的高级解决方案方面的知识。此外，该项目支持加州大学戴维斯分校的多样性和高等教育，同时通过将研究融入加州大学戴维斯分校的教学中来改善教育。最终，该项目的成功将使各种应用更容易获得优质变压器模型，从而造福社会。研究人员在他们的新变压器模型中探索了一种增量采样方法，以跨编码器层处理输入图像，逐步获得上下文感知。他们的目标是利用增量上下文感知来删除无人看管的令牌，并掩盖新样本中不重要的输入补丁。此外，研究人员还探索了基于学习和上下文感知的注意头下降、编码器层跳过和粗粒模型修剪的早期终止。为了提高变压器模型的推理效率，研究人员探索构建一个随机预处理单元，该单元近似于矩阵-矩阵乘法，支持基于注意力的模型修剪分类器，用于patch， token，注意力头和编码器消除。为了构建硬件加速器的乘法和累积（MAC）单元，研究人员探索了一种新的时间进位延迟解决方案，消除了MAC中的进位传播。该解决方案简化了MAC逻辑，提高了流处理速度和效率。此外，研究人员的目标是利用新MAC的浅逻辑深度来设计高效的扩散性MAC，从而在处理阵列中实现动态精度权衡。研究人员还研究开发一种调度器，用于平衡工作负载，并在操作稀疏注意图时最大限度地减少内存访问，支持乱序令牌处理、处理元素聚类、顺序完成和精度感知调度，以优化性能。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

Transformer models are a relatively recent breakthrough in machine learning that have revolutionized natural language processing and boosted the generalization of computer vision models. However, the wide adoption of transformer models requires making them significantly more energy efficient. The Transformer models are too complex, and existing hardware is not optimal for their efficient execution. In this project, researchers are exploring two interrelated research problems to tackle transformer efficiency: 1) creating new Transformer models that can be dynamically pruned to improve efficiency without sacrificing accuracy, and 2) designing specialized hardware to make the execution of Transformers more efficient. The impact of this project is significant in several ways: Firstly, it promotes scientific progress in the research community by advancing communities understanding of attention as the primary mechanism in transformer models and how it can be used for context-aware pruning of complex learning models. Secondly, it extends knowledge of the hardware community in designing advanced solutions for dynamic precision tuning and scheduling of complex and tunable hardware systems. Additionally, the project supports diversity and higher education at University of California (UC) – Davis while improving education by integrating research into teaching at UC Davis classes. Ultimately, the project's success will benefit society by making superior transformer models more accessible across various applications.Researchers explore an incremental sampling approach in their new transformer model to process input images across encoder layers gaining contextual awareness progressively. They aim to leverage incremental contextual awareness to remove unattended tokens and mask unimportant input patches in new samples. Additionally, researchers explore learning-based and context-aware attention-head dropping, encoder-layer skipping, and early termination for coarse grain model pruning. To improve the transformer model's inference efficiency, the researchers explore architecting a stochastic pre-processing unit that approximates matrix-matrix multiplication supporting attention-based model pruning classifiers for patch, token, attention head, and encoder elimination. To build the hardware accelerator's multiplication and accumulation (MAC) units, researchers explore a novel solution for temporal carry-bit deferment, eliminating carry-bit propagation in MAC. This solution simplifies MAC logic, enhancing stream processing speed and efficiency. Furthermore, Researchers aim to leverage the shallow logic depth of the new MAC to design highly efficient diffusible MACs, enabling dynamic precision trade-offs in the processing array. Researchers also investigate developing a scheduler for balancing workload and minimizing memory accesses when operating on sparse attention graphs with support for out-of-order token processing, processing element clustering, in-order completion, and precision-aware scheduling to optimize performance.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.