AitF: Collaborative Research: A Framework of Simultaneous Acceleration and Storage Reduction on Deep Neural Networks Using Structured Matrices

深度神经网络（dnn）已经成为一种强大的技术，用于在许多具有挑战性的问题领域学习解决方案，包括计算机视觉，自然语言处理和生物信息学。这些解决方案之所以能够实现，主要是因为我们现在有了计算加速器，能够筛选训练神经网络所需的无数数据。随着深度神经网络模型的规模不断增长，训练所需的计算和内存资源也将增长，这限制了深度学习在许多实际应用中的部署。利用结构化矩阵理论，该项目将为有效的深度神经网络训练和推理开发一个通用框架，在计算和存储方面显著降低算法复杂性。该项目如果成功，将从根本上影响广泛的深度学习应用。它将探索加速这种针对新兴加速器架构的深度学习算法的新结构，并将评估这些进步在许多应用领域的好处，包括大数据分析、认知系统、无人驾驶车辆和空中系统以及可穿戴设备。该项目的跨学科性质连接了矩阵理论、机器学习和计算机体系结构等领域，并将影响东北大学和CCNY的教育，包括让代表性不足的本科生参与丰富的研究任务。该项目将：(1)首次开发基于结构化矩阵的深度神经网络模型的一般理论框架，并对误差界、收敛性、快速训练算法等进行详细分析和研究；(2)为dnn的全连接层开发低空间成本和高速的推理和训练方案；(3)施加具有结构的权张量，实现低计算和空间成本的卷积层；(4)在高性能并行平台、低功耗嵌入式平台以及新兴计算范式和设备上开发高效节能的深度学习系统实现；(5)对各种平台上不同性能指标的拟议方法进行综合评估。该项目将提供针对一系列计算平台的优化实现，包括asic、fpga、gpu和云服务器。硬件优化将专注于生产高速、低成本的深度学习系统实现。

Deep neural networks (DNNs) have emerged as a class of powerful techniques for learning solutions in a number of challenging problem domains, including computer vision, natural language processing and bioinformatics. These solutions have been enabled mainly because we now have computational accelerators able to sift through the myriad of data required to train a neural network. As the size of DNN models continues to grow, computational and memory resource requirements for training will also grow, limiting deployment of deep learning in many practical applications. Leveraging the theory of structured matrices, this project will develop a general framework for efficient DNN training and inference, providing a significant reduction in algorithmic complexity measures in terms of both computation and storage. The project, if successful, should fundamentally impact a broad class of deep learning applications. It will explore accelerating this new structure for deep learning algorithms targeting emerging accelerator architectures, and will evaluate the benefits of these advances across a number of application domains, including big data analytics, cognitive systems, unmanned vehicles and aerial systems, and wearable devices. The interdisciplinary nature of this project bridges the areas of matrix theory, machine learning, and computer architecture, and will affect education at both Northeastern and CCNY, including the involvement of underrepresented and undergraduate students in the rich array of research tasks.     The project will: (1) for the first time, develop a general theoretical framework for structured matrix-based DNN models and perform detailed analysis and investigation of error bounds, convergence, fast training algorithms, etc.; (2) develop low-space-cost and high-speed inference and training schemes for the fully connected layers of DNNs; (3) impose a weight tensor with structure and enable low computational and space cost convolutional layers; (4) develop high-performance and energy-efficient implementations of deep learning systems on high-performance parallel platforms, low-power embedded platforms, as well as emerging computing paradigms and devices; (5) perform a comprehensive evaluation of the proposed approaches on different performance metrics in a variety of platforms.  The project will deliver tuned implementations targeting a range of computational platforms, including ASICs, FPGAs, GPUs and cloud servers. The hardware optimizations will focus on producing high-speed and low-cost implementations of deep learning systems.