CAREER: Differentiable Network-Accelerator Co-Search Towards Ubiquitous On-Device Intelligence and Green AI

在强大的深度学习(DL)算法令人望而却步的复杂性和可用于实施它们的有限资源之间存在着巨大且不断扩大的差距。最近人们认识到，联合设计DL算法及其硬件加速器在缩小这一巨大差距方面非常有前途。然而，现有的作品才刚刚开始触及其全部潜力的表面。该项目旨在通过联合搜索(联合搜索)DL算法及其加速器来促进在开发DL加速器及其可实现的加速效率方面的系统性突破。该项目的总体目标是开发、实施和实验验证一种设计深度数字减速加速器的新范例，以实现：(1)更快的开发速度；(2)更高的硬件效率；(3)前所未有的灵活性来控制硬件效率和任务性能之间的权衡，通过全面促进自动化网络加速器协同搜索的系统性突破。教育计划是继续并扩大与全民科技的现有合作，这是一个针对低收入和服务不足的人的组织，通过指导和建议来自代表性不足社区的高中生。拟议的研究将促进知识的进步，并为设计DL加速器的新范式提供科学的原理和工具，使开发速度和硬件效率都有数量级的改进。首先，将开发通用设计空间描述和性能预测器，作为(1)自动加速器搜索和(2)自动网络加速器协同搜索的关键推动因素，为创新高效的DL加速器开辟许多机会。其次，基于上述设计空间描述和性能预测器，将设计一个自动化和可区分硬件加速器搜索(D-HAS)引擎，以实现(1)在DL加速器的大而离散的设计空间上的高效导航，以及(2)显著加快DL加速器的开发。第三，在上述D-HAS的基础上，将建立一个创新的网络-加速器协同搜索框架，以支持同时搜索最优的DL网络和加速器对，共同最大限度地提高可实现的硬件效率。最后，一个独特的资源(莱斯大学的ASTRO平台)将被用来对创新进行基准测试和展示。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

There exists a vast and increasing gap between the prohibitive complexity of powerful Deep Learning (DL) algorithms and the constrained resources available for implementing them. It has been recently recognized that jointly designing the DL algorithms and their hardware accelerators is very promising in closing this vast gap. However, existing works have just begun to scratch the surface of its full potential. This project aims to foster a systematic breakthrough in developing DL accelerators and their achievable acceleration efficiency by jointly searching (co-search) for DL algorithms and their accelerators. The overarching goal of this project is to develop, implement, and experimentally validate a new paradigm of designing deep DL accelerators for enabling: (1) orders of magnitude faster development speed; (2) much improved hardware efficiency, and (3) unprecedented flexibility to control the trade-off between hardware efficiency and task performance, by holistically fostering a systematic breakthrough in automated network-accelerator co-search. The educational plan is to continue and expand an existing collaboration with Technology for All, an organization that targets low-income and underserved persons, by mentoring and advising high school students from underrepresented communities. The proposed research will advance knowledge and produce scientific principles and tools for a new paradigm of designing DL accelerators with orders-of-magnitude improvement in both development speed and hardware efficiency. First, a generic design space description and a performance predictor will be developed to serve as key enablers for both (1) automated accelerator search and (2) automated network-accelerator co-search, opening up many opportunities for innovating efficient DL accelerators. Second, based on the aforementioned design space description and performance predictor, an automated and Differentiable Hardware Accelerator Search (D-HAS) engine will be designed to enable both (1) efficient navigation over the large and discrete design space of DL accelerators and (2) the significantly faster development of DL accelerators. Third, building upon the above D-HAS, an innovative network-accelerator co-search framework will be established to enable simultaneous search for optimal DL network and accelerator pairs that together will maximize the achievable hardware efficiency. Finally, a unique resource (Rice University's ASTRO platform) will be leveraged to benchmark and demonstrate the innovations.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.