CAREER: Differentiable Network-Accelerator Co-Search Towards Ubiquitous On-Device Intelligence and Green AI

职业生涯：可微分网络加速器联合搜索，实现无处不在的设备智能和绿色人工智能

• 批准号: 2048183
• 负责人: Yingyan Lin
• 金额: $ 40万
• 依托单位: William Marsh Rice University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-04-01 至 2023-11-30
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2048183&HistoricalAwards=false
• 关键词: CAREER; Differentiable; Network; Accelerator; Co

There exists a vast and increasing gap between the prohibitive complexity of powerful Deep Learning (DL) algorithms and the constrained resources available for implementing them. It has been recently recognized that jointly designing the DL algorithms and their hardware accelerators is very promising in closing this vast gap. However, existing works have just begun to scratch the surface of its full potential. This project aims to foster a systematic breakthrough in developing DL accelerators and their achievable acceleration efficiency by jointly searching (co-search) for DL algorithms and their accelerators. The overarching goal of this project is to develop, implement, and experimentally validate a new paradigm of designing deep DL accelerators for enabling: (1) orders of magnitude faster development speed; (2) much improved hardware efficiency, and (3) unprecedented flexibility to control the trade-off between hardware efficiency and task performance, by holistically fostering a systematic breakthrough in automated network-accelerator co-search. The educational plan is to continue and expand an existing collaboration with Technology for All, an organization that targets low-income and underserved persons, by mentoring and advising high school students from underrepresented communities. The proposed research will advance knowledge and produce scientific principles and tools for a new paradigm of designing DL accelerators with orders-of-magnitude improvement in both development speed and hardware efficiency. First, a generic design space description and a performance predictor will be developed to serve as key enablers for both (1) automated accelerator search and (2) automated network-accelerator co-search, opening up many opportunities for innovating efficient DL accelerators. Second, based on the aforementioned design space description and performance predictor, an automated and Differentiable Hardware Accelerator Search (D-HAS) engine will be designed to enable both (1) efficient navigation over the large and discrete design space of DL accelerators and (2) the significantly faster development of DL accelerators. Third, building upon the above D-HAS, an innovative network-accelerator co-search framework will be established to enable simultaneous search for optimal DL network and accelerator pairs that together will maximize the achievable hardware efficiency. Finally, a unique resource (Rice University's ASTRO platform) will be leveraged to benchmark and demonstrate the innovations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

强大的深度学习（DL）算法的复杂性与可用于实现它们的有限资源之间存在巨大且不断增长的差距。最近已经认识到，联合设计DL算法及其硬件加速器在缩小这一巨大差距方面非常有前途。然而，现有的作品才刚刚开始触及其全部潜力的表面。该项目旨在通过联合搜索（共同搜索）DL算法及其加速器，在开发DL加速器及其可实现的加速效率方面取得系统性突破。该项目的总体目标是开发，实施和实验验证设计深度DL加速器的新范式，以实现：（1）数量级更快的开发速度;（2）大大提高硬件效率，以及（3）前所未有的灵活性，以控制硬件效率和任务性能之间的权衡，通过全面促进自动化网络加速器共同搜索的系统性突破。教育计划将继续并扩大与面向低收入和得不到充分服务的人的组织“人人享有技术”的现有合作，为来自代表性不足社区的高中生提供辅导和咨询。拟议的研究将推进知识，并产生科学的原则和工具，为设计DL加速器的新范式，在开发速度和硬件效率方面都有数量级的改进。首先，将开发通用设计空间描述和性能预测器，以作为（1）自动化加速器搜索和（2）自动化网络加速器共同搜索的关键推动因素，为创新高效DL加速器提供许多机会。其次，基于上述设计空间描述和性能预测器，将设计自动化和可区分的硬件加速器搜索（D-HAS）引擎，以实现（1）在DL加速器的大型离散设计空间上的有效导航和（2）DL加速器的显著更快的开发。第三，在上述D-HAS的基础上，将建立创新的网络加速器共同搜索框架，以使得能够同时搜索最佳DL网络和加速器对，其一起将最大化可实现的硬件效率。最后，一个独特的资源（莱斯大学的ASTRO平台）将被用来衡量和展示创新。该奖项反映了NSF的法定使命，并被认为是值得通过使用基金会的知识价值和更广泛的影响审查标准进行评估的支持。

项目成果

Contrastive quant: quantization makes stronger contrastive learning

• DOI: 10.1145/3489517.3530419
• 发表时间: 2022-07
• 期刊: Proceedings of the 59th ACM/IEEE Design Automation Conference
• 作者: Y. Fu;Qixuan Yu;Meng Li;Xuefeng Ouyang;Vikas Chandra;Yingyan Lin

Auto-NBA: Efficient and Effective Search Over the Joint Space of Networks, Bitwidths, and Accelerators

Auto-NBA：网络、比特宽度和加速器联合空间的高效搜索

• DOI: 10.48550/arxiv.2106.06575
• 发表时间: 2021
• 期刊: Proceedings of the 38th International Conference on Machine Learning (ICML 2021
• 作者: Fu, Yonggan;Zhang, Yongan;Zhang, Yang;Cox, David;Lin, Yingyan
• 通讯作者: Lin, Yingyan

A3C-S: Automated Agent Accelerator Co-Search towards Efficient Deep Reinforcement Learning

A3C-S：自动化代理加速器协同搜索，实现高效深度强化学习

• DOI: 10.1109/dac18074.2021.9586305
• 发表时间: 2021
• 期刊: 2021 58th ACM/IEEE Design Automation Conference (DAC
• 作者: Fu, Yonggan;Zhang, Yongan;Li, Chaojian;Yu, Zhongzhi;Lin, Yingyan
• 通讯作者: Lin, Yingyan

ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attention

• DOI: 10.1109/hpca56546.2023.10071081
• 发表时间: 2022-11
• 期刊: 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA)
• 作者: Jyotikrishna Dass;Shang Wu;Huihong Shi;Chaojian Li;Zhifan Ye;Zhongfeng Wang;Yingyan Lin

Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?

• DOI: 10.48550/arxiv.2203.08392
• 发表时间: 2022-03
• 期刊: ArXiv
• 作者: Y. Fu;Shunyao Zhang;Shan-Hung Wu;Cheng Wan;Yingyan Lin