SHF:Small: Tensor-Based Algorithm and Hardware Co-Optimization for Neural Network Architecture

SHF:Small：基于张量的神经网络架构算法和硬件协同优化

• 批准号: 1817037
• 负责人: Zheng Zhang
• 金额: $ 50万
• 依托单位: University of California-Santa Barbara
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-10-01 至 2022-03-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1817037&HistoricalAwards=false
• 关键词: SHF; Small; Tensor; Based; Algorithm

Machine learning plays an important role in our daily life, including medical data analysis, finance, autonomous driving and computer vision. Many popular machine learning models are both data-intensive and computationally expensive. In order to address this challenge, algorithm and architecture co-design and co-optimizations are required to achieve better performance and energy efficiency. This project aims to develop a more powerful learning architecture by simultaneously optimizing the neural network algorithm and its hardware implementation. The project will have a broad impact. The AI applications will gain a significant performance boost if the computational and storage cost of its algorithm can be reduced significantly. It will improve many applications such as data mining, medical imaging analysis, computational biology, and financial analysis. The project will greatly enrich the undergraduate and graduate course curriculum and attract graduate and undergraduate students to participate in this project and related workshops. Finally, this project will develop new AI, VLSI and computer architecture workforce with solid background in several areas including computational math, machine learning, and hardware design.Tensors are a generalization of vectors and matrices, and they are promising tools to represent and numerically process high-dimensional data arrays. Leveraging the high effectiveness of tensor computation in big-data analysis, this project will investigate three specific topics towards designing high-performance and energy-efficient machine learning hardware. First, theoretically sound and novel tensor numerical algorithms will be developed to significantly reduce the training and inference cost of deep learning. Second, the algorithm framework will be optimized on existing hardware (e.g., GPU and FPGA) to achieve better performance, and to examine the main challenges when running on hardware platforms. Finally, emerging design technologies (e.g., 3-D process-in-memory) will be investigated to design specific hardware libraries to perform fundamental tensor computation and to further boost the performance and energy efficiency of the whole machine learning architecture.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

机器学习在我们的日常生活中扮演着重要的角色，包括医疗数据分析、金融、自动驾驶和计算机视觉。许多流行的机器学习模型都是数据密集型的，计算成本也很高。为了应对这一挑战，需要算法和架构协同设计和协同优化，以实现更好的性能和能源效率。本项目旨在通过同时优化神经网络算法及其硬件实现来开发更强大的学习架构。这个项目将产生广泛的影响。如果算法的计算和存储成本能够显著降低，人工智能应用程序将获得显著的性能提升。它将改进许多应用，如数据挖掘、医学成像分析、计算生物学和金融分析。该项目将极大地丰富本科和研究生课程设置，吸引研究生和本科生参与本项目和相关研讨会。最后，该项目将开发新的人工智能、超大规模集成电路和计算机架构劳动力，他们在计算数学、机器学习和硬件设计等多个领域拥有扎实的背景。张量是向量和矩阵的泛化，它们是表示和数值处理高维数据数组的有前途的工具。利用张量计算在大数据分析中的高效率，该项目将研究设计高性能和节能机器学习硬件的三个具体主题。首先，理论上健全和新颖的张量数值算法将被开发出来，以显着降低深度学习的训练和推理成本。其次，算法框架将在现有硬件（例如GPU和FPGA）上进行优化，以获得更好的性能，并检查在硬件平台上运行时的主要挑战。最后，将研究新兴的设计技术（例如，内存中的3-D进程），以设计特定的硬件库来执行基本张量计算，并进一步提高整个机器学习架构的性能和能效。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

项目成果

Fast Search of the Optimal Contraction Sequence in Tensor Networks

张量网络中最优收缩序列的快速搜索

• DOI: 10.1109/jstsp.2021.3051231
• 发表时间: 2021
• 期刊: IEEE Journal of Selected Topics in Signal Processing
• 影响因子: 7.5
• 作者: Liang, Ling;Xu, Jianyu;Deng, Lei;Yan, Mingyu;Hu, Xing;Zhang, Zheng;Li, Guoqi;Xie, Yuan
• 通讯作者: Xie, Yuan

Hardware-Enabled Efficient Data Processing With Tensor-Train Decomposition

• DOI: 10.1109/tcad.2021.3058317
• 发表时间: 2021-02
• 期刊: IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
• 影响因子: 2.9
• 作者: Zheng Qu;Bangyan Wang;Hengnu Chen;Jilan Lin;Ling Liang;Guoqi Li;Zheng Zhang;Yuan Xie

Sparse Tucker Tensor Decomposition on a Hybrid FPGA–CPU Platform

混合 FPGA-CPU 平台上的稀疏 Tucker 张量分解

• DOI: 10.1109/tcad.2020.3032626
• 发表时间: 2021
• 期刊: IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
• 影响因子: 2.9
• 作者: Jiang, Weiyun;Zhang, Kaiqi;Lin, Colin Yu;Xing, Feng;Zhang, Zheng
• 通讯作者: Zhang, Zheng

Bayesian tensorized neural networks with automatic rank selection

• DOI: 10.1016/j.neucom.2021.04.117
• 发表时间: 2021-05-19
• 期刊: NEUROCOMPUTING
• 影响因子: 6
• 作者: Hawkins, Cole;Zhang, Zheng
• 通讯作者: Zhang, Zheng

On-FPGA Training with Ultra Memory Reduction: A Low-Precision Tensor Method

• 发表时间: 2021-04
• 期刊: ArXiv
• 作者: Kaiqi Zhang;Cole Hawkins;Xiyuan Zhang;Cong Hao;Zheng Zhang