Collaborative Research: SHF: Medium: Analog EDA-Inspired Methods for Efficient and Robust Neural Network Design

深度神经网络在许多工程领域取得了巨大成功，包括但不限于图像分类、语音识别、推荐系统和自动驾驶。然而，它们面临两大挑战。首先，许多神经网络模型不鲁棒，即当输入数据经历非常少量的扰动时，神经网络可能产生不准确的结果。其次，生成和部署大型神经网络的巨大成本限制了它们在资源受限平台（例如移动的设备和机器人）中的应用。研究小组注意到，某些类型的神经网络和模拟集成电路之间存在很强的数学联系。众所周知，EDA（电子设计自动化）领域在模拟集成电路的建模、仿真、验证和优化方面已经有50年的成功历史。因此，该项目旨在通过利用EDA社区中的原则性方法，大大丰富神经网络的算法和理论理解。这项研究将支持在圣巴巴拉的加州大学，圣地亚哥的加州大学和马萨诸塞州理工学院的研究生和本科生的不同队列的跨学科发展。正在创建或充实关于计算方法、数据科学和人工智能的几门研究生课程。威利斯的研究团队还与工业界合作，以确保有效的技术转让。该项目侧重于某些类型的深度神经网络（例如，残差神经网络、递归神经网络和归一化流），其可以被描述为常微分方程。该项目的技术目标分为三个方面。第一个重点是从电路仿真和建模的角度研究深度神经网络的训练和压缩算法。具体而言，并行训练算法正在开发的神经网络借用的想法，从并行电路模拟。硬件友好的神经网络压缩算法正在从电路模型降阶的角度开发，从而实现深度神经网络的节能和实时推理。第二个重点是从电路不确定性量化的角度研究深度神经网络鲁棒性的概率和准确验证技术。具体而言，正在开发高置信度和更严格的验证边界，以通过利用模拟电路不确定性量化中的分层和非蒙特卡罗技术来描述深度神经网络的可达集。第三个目标是从模拟电路成品率优化的角度提高深度神经网络的鲁棒性。在这最后的推力，两个想法正在探索：（1）前硅产量优化技术的鲁棒神经网络的训练，和（2）后硅自我修复技术的鲁棒性提高训练的神经网络。这个奖项反映了NSF的法定使命，并已被认为是值得通过评估使用基金会的智力价值和更广泛的影响审查标准的支持。

Deep neural networks have achieved great success in many engineering fields including, but not limited to, image classification, speech recognition, recommendation systems and autonomous driving. However, they suffer from two major challenges. Firstly, many neural network models are not robust, i.e. a neural network could produce inaccurate results when the input data experiences a very small amount of perturbation. Secondly, the huge cost of generating and deploying large-size neural networks limits their applications in resource-constrained platforms (e.g. mobile devices and robots). The research team notices that there is a strong mathematical connection between certain types of neural networks and analog integrated circuits. It is also known that the EDA (electronic design automation) field has 50 years of successful history of modeling, simulating, verifying and optimizing analog integrated circuits. Therefore, this project aims to substantially enrich the algorithms and theoretical understanding of neural networks by leveraging the principled approaches in the EDA community. This research will support the cross-disciplinary development of a diverse cohort of graduate and undergraduate students at the University of California at Santa Barbara, the University of California at San Diego, and the Massachusetts Institute of Technology. Several graduate-level courses on computational methods, data science and artificial intelligence are being created or enriched. The research team willis also collaborating with industry to ensure effective technology transfers.This project focuses on certain types of deep neural networks (e.g., residual neural networks, recurrent neural networks and normalizing flows) that can be described as ordinary differential equations. The technical aims of the project are divided into three thrusts. The first thrust investigates the training and compression algorithms of deep neural networks from circuit simulation and modeling perspectives. Specifically, parallel training algorithms are being developed for neural networks by borrowing the idea from parallel circuit simulation. Hardware-friendly neural-network compression algorithms are being developed from the perspective of circuit model order reduction, thereby enabling energy-efficient and real-time inference of deep neural networks. The second thrust investigates probabilistic and accurate verification techniques for the robustness of deep neural networks from circuit uncertainty quantification perspectives. Specifically, high-confidence and tighter verification bounds are being developed to describe the reachable set of a deep neural network by leveraging the hierarchical and non-Monte-Carlo techniques in analog circuit uncertainty quantification. The third thrust aims to improve the robustness of a deep neural network from the perspective of analog circuit yield optimization. In this final thrust, two ideas are being explored: (1) pre-silicon yield optimization techniques for robust neural network training, and (2) post-silicon self-healing techniques for robustness improvement of a trained neural network.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.