AitF: Collaborative Research: A Framework of Simultaneous Acceleration and Storage Reduction on Deep Neural Networks Using Structured Matrices

AitF：协作研究：使用结构化矩阵的深度神经网络同时加速和存储减少的框架

• 批准号: 1854742
• 负责人: Bo Yuan
• 金额: $ 36.79万
• 依托单位: Rutgers University New Brunswick
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2018
• 资助国家: 美国
• 起止时间: 2018-08-15 至 2021-12-31
• 项目状态: 已结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=1854742&HistoricalAwards=false
• 关键词: AitF; Collaborative; Research; Framework; Simultaneous

Deep neural networks (DNNs) have emerged as a class of powerful techniques for learning solutions in a number of challenging problem domains, including computer vision, natural language processing and bioinformatics. These solutions have been enabled mainly because we now have computational accelerators able to sift through the myriad of data required to train a neural network. As the size of DNN models continues to grow, computational and memory resource requirements for training will also grow, limiting deployment of deep learning in many practical applications. Leveraging the theory of structured matrices, this project will develop a general framework for efficient DNN training and inference, providing a significant reduction in algorithmic complexity measures in terms of both computation and storage. The project, if successful, should fundamentally impact a broad class of deep learning applications. It will explore accelerating this new structure for deep learning algorithms targeting emerging accelerator architectures, and will evaluate the benefits of these advances across a number of application domains, including big data analytics, cognitive systems, unmanned vehicles and aerial systems, and wearable devices. The interdisciplinary nature of this project bridges the areas of matrix theory, machine learning, and computer architecture, and will affect education at both Northeastern and CCNY, including the involvement of underrepresented and undergraduate students in the rich array of research tasks.     The project will: (1) for the first time, develop a general theoretical framework for structured matrix-based DNN models and perform detailed analysis and investigation of error bounds, convergence, fast training algorithms, etc.; (2) develop low-space-cost and high-speed inference and training schemes for the fully connected layers of DNNs; (3) impose a weight tensor with structure and enable low computational and space cost convolutional layers; (4) develop high-performance and energy-efficient implementations of deep learning systems on high-performance parallel platforms, low-power embedded platforms, as well as emerging computing paradigms and devices; (5) perform a comprehensive evaluation of the proposed approaches on different performance metrics in a variety of platforms.  The project will deliver tuned implementations targeting a range of computational platforms, including ASICs, FPGAs, GPUs and cloud servers. The hardware optimizations will focus on producing high-speed and low-cost implementations of deep learning systems.

深度神经网络（DNN）已经成为一类强大的技术，用于在许多具有挑战性的问题领域学习解决方案，包括计算机视觉，自然语言处理和生物信息学。 这些解决方案之所以能够实现，主要是因为我们现在有了计算加速器，能够筛选训练神经网络所需的大量数据。 随着DNN模型的规模不断增长，用于训练的计算和内存资源需求也将增长，这限制了深度学习在许多实际应用中的部署。 利用结构化矩阵理论，该项目将开发一个有效DNN训练和推理的通用框架，在计算和存储方面显著降低算法复杂性。 该项目如果成功，将从根本上影响广泛的深度学习应用。 它将探索加速这种针对新兴加速器架构的深度学习算法的新结构，并将评估这些进步在许多应用领域的好处，包括大数据分析，认知系统，无人驾驶车辆和航空系统以及可穿戴设备。 该项目的跨学科性质桥接了矩阵理论，机器学习和计算机体系结构的领域，并将影响东北大学和CCNY的教育，包括代表性不足和本科生参与丰富的研究任务。 该项目将：（1）首次提出了基于结构矩阵的DNN模型的一般理论框架，并对误差界、收敛性、快速训练算法等进行了详细的分析和研究; (2)为DNN的全连接层开发低空间成本和高速推理和训练方案;（3）施加具有结构的权重张量，并实现低计算和空间成本的卷积层;（4）在高性能并行平台，低功耗嵌入式平台以及新兴计算范式和设备上开发高性能和节能的深度学习系统实现;（5）在不同的平台上对所提出的方法进行全面评估。该项目将针对一系列计算平台（包括ASIC、FPGA、GPU和云服务器）提供优化的实现。硬件优化将专注于深度学习系统的高速和低成本实现。

项目成果

CircConv: A Structured Convolution with Low Complexity

• DOI: 10.1609/aaai.v33i01.33014287
• 发表时间: 2019-02
• 期刊: ArXiv
• 作者: Siyu Liao;Zhe Li;Liang Zhao;Qinru Qiu;Yanzhi Wang;Bo Yuan

New Practical Advances in Polynomial Root Clustering

多项式根聚类的新实用进展

• DOI: 10.1007/978-3-030-43120-4_11
• 发表时间: 2019
• 期刊: Mathematical Aspects of Computer and Information Sciences (MACIS 2019
• 作者: Imbach, R;Pan, V
• 通讯作者: Pan, V

Old and New Nearly Optimal Polynomial Root-Finders

新旧近乎最优多项式求根器

• DOI: 10.1007/978-3-030-26831-2_26
• 发表时间: 2019
• 期刊: 21st International Workshop on Computer Algebra in Scientific Computing (CASC'2019
• 作者: Pan, Victor

Sublinear Cost Low Rank Approximation via Subspace Sampling

通过子空间采样的次线性成本低阶近似

• DOI: 10.1007/978-3-030-43120-4_9
• 发表时间: 2019
• 期刊: Mathematical Aspects of Computer and Information Sciences
• 作者: Pan, V;Luan, Q;Svadlenka, J;Zhao, L
• 通讯作者: Zhao, L

CUR Low Rank Approximation at Sub-linear Cost

次线性成本下的 CUR 低秩近似

• 发表时间: 2019
• 期刊: ArXiv.org
• 作者: Pan, Victor Y;Luan, Q;Svadlenka, J;Zhao, L.
• 通讯作者: Zhao, L.