EAGER: IMPRESS-U: Random Matrix Theory and its Applications to Deep Learning

这个IMPRESS-U项目将由美国国家科学基金会、美国国家科学院和波兰国家科学中心联合支持。这项研究将与美国宾夕法尼亚州立大学、乌克兰国家科学院低温物理与工程研究所(ILTPE)和波兰华沙大学合作进行。IMPRESS-U项目的美国部分是由国际科学与工程办公室和MPS/DMS应用数学和计算数学项目共同资助的。第一部分人工智能的主要思想形成于20世纪70年代，当时化学、汽车和其他行业的复杂工艺流程变得依赖于大量随机和非随机因素。为了应对这一挑战，人们提出了自主学习系统(SLS)。SLS允许对生产线进行微调，并适应不断变化的条件(温度、压力等)。深度神经网络(DNN)是SLS最有前途的现代实现之一，它目前被广泛使用，从工业和国防系统到互联网技术和手机。DNN学习过程的一个关键部分是调整DNN的参数以提高其性能。这些参数形成具有典型初始随机条目的矩阵。因此，随机矩阵理论(RMT)的应用可以极大地促进学习过程。在这个项目中，预计将通过开发RMT工具来建立优化学习的标准。因此，可以加快学习过程，提高DNN的准确性，降低复杂性，避免过度训练。该项目涉及来自美国(宾夕法尼亚州立大学)、波兰和乌克兰的研究团队。该项目将在很大程度上建立在乌克兰哈尔科夫数学学院在RMT方面的世界级实力。该项目将帮助哈尔科夫的研究生、博士后和研究人员融入国际科学界和研究队伍，并培养在STEM领域工作的新一代乌克兰研究人员。乌克兰学生将了解RMT在DNN中的最新应用。受生物神经网络功能的启发，深度神经网络(DNN)在对象、语音和模式识别等广泛的尖端应用中展示了其高效性。然而，从理论的角度对它们的理解仍然很少。最近的研究表明，基于随机矩阵理论(RMT)的分析有助于提高神经网络的收敛速度和学习速度，并在深度学习中提高训练算法的精度和降低计算复杂度。该项目的重点有两个：(1)进一步发展RMT技术，(2)利用这些技术发展深度学习中的数值和分析方法。具体地说，将从解析和数值两方面研究在DNN分析中产生的随机矩阵系综的谱特性。特别是，该项目将通过基于RMT的修剪和正则化技术来提高DNN的准确性并降低计算复杂性。为此，该研究将致力于为未训练的DNN寻找最优的权值初始化，确定训练的停止标准，研究非线性对DNN学习速度的影响，以及验证数值和近似方法以及误差估计。该项目汇集了数学和物理领域的不同领域和途径的研究人员。这些领域从深度学习、机器学习、RMT和凝聚态物理到概率论和分析。该项目将开发RMT和深度学习中的分析和数值工具，并丰富全球数学界。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This IMPRESS-U project will be jointly supported by NSF, US National Academy of Sciences, and National Science Centre of Poland. The research will be conducted in collaborative partnership that unites the Pennsylvania State University, USA, Institute for Low Temperature Physics and Engineering of the National Academy of Sciences of Ukraine (ILTPE), and the University of Warsaw, Poland. The USA part of this IMPRESS-U project is co-funded by the Office of International Science and Engineering and MPS/DMS Applied Mathematics and Computational Mathematics programs.Part 1.The main ideas of Artificial Intelligence were formulated in 1970s, when complex technological processes in chemical, automobile and other industries became dependent on an enormous number of random and non-random factors. To answer this challenge, self-learning systems (SLS) were proposed. The SLS allow for fine tuning of production lines and adapting to constantly changing conditions (temperature, pressure, etc). One of the most promising modern realizations of SLS is Deep Neural Networks (DNNs) which are currently used everywhere from industry and defense systems up to internet technologies and cell phones. A key component of the DNNs’ learning process is adjusting the parameters of a DNN to increase its performance. These parameters form matrices with typically initial random entries. Hence, the application of Random Matrix Theory (RMT) could greatly benefit the learning process. In this project, it is expected to establish criteria for optimal learning via developing RMT tools. As a result, it will be possible to speed up the learning process, increase the DNN accuracy, reduce complexity, and avoid over-training.The project engages research teams from the US (Penn State University), Poland, and Ukraine. The project will build to a large extent upon the world-class strength in RMT of the school of mathematics in Kharkiv, Ukraine. The project will help to integrate graduate students, postdocs, and researchers from Kharkiv into the international scientific community and research workforce, and to prepare a new generation of Ukrainian researchers working in STEM fields. The Ukrainian students will be introduced to the state-of-the-art applications of RMT in DNNs.Part 2.Inspired by the function of biological neural networks, Deep Neural Networks (DNNs) have demonstrated their high effectiveness in a wide range of cutting-edge applications such as object, speech, and pattern recognition. However, they are still poorly understood from a theoretical point of view. Recent studies have shown that analysis based on Random Matrix Theory (RMT) can help to improve the convergence and learning speed of neural networks as well as improve accuracy and reduce the computational complexity of training algorithms in deep learning. The focus of the project is twofold: (i) further development of RMT techniques, (ii) employment of these techniques for developing numerical and analytical methods in deep learning. Specifically, the spectral properties of random matrix ensembles arising in the analysis of DNNs will be studied, both analytically and numerically. In particular, the project will address enhancing accuracy of DNNs and reducing computational complexity via RMT-based pruning and regularization techniques. To this end, the research will be concerned with finding an optimal weight initialization for untrained DNNs, determining stopping criteria for training, studying the nonlinearity’s effect on DNN learning speed, as well as justifying numerical and approximation methods and error estimates.The project brings together researchers from various fields and approaches in mathematics and physics. These fields range from deep learning, machine learning, RMT, and condensed matter physics to probability theory, and analysis. The project will develop analytical and numerical tools in RMT and deep learning and enrich the global mathematical community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.