EAGER: IMPRESS-U: Random Matrix Theory and its Applications to Deep Learning

该IMPRESS-U项目将由NSF、美国国家科学院和波兰国家科学中心共同支持。该研究将在合作伙伴关系中进行，该合作伙伴关系将美国宾夕法尼亚州立大学，乌克兰国家科学院低温物理与工程研究所（ILTPE）和波兰华沙大学联合起来。IMPRESS-U项目的美国部分由国际科学与工程办公室和MPS/DMS应用数学和计算数学项目共同资助。第一部分人工智能的主要思想形成于20世纪70年代，当时化学，汽车和其他行业的复杂工艺过程变得依赖于大量的随机和非随机因素。为了应对这一挑战，提出了自学习系统（SLS）。SLS允许对生产线进行微调，并适应不断变化的条件（温度、压力等）。SLS最有前途的现代实现之一是深度神经网络（DNN），目前从工业和国防系统到互联网技术和手机，都在使用它。DNN学习过程的一个关键组成部分是调整DNN的参数以提高其性能。这些参数形成通常具有初始随机条目的矩阵。因此，随机矩阵理论（RMT）的应用可以大大有利于学习过程。在这个项目中，预计将通过开发RMT工具建立最佳学习的标准。因此，它将有可能加快学习过程，提高DNN的准确性，降低复杂性，并避免过度训练。该项目涉及来自美国（宾夕法尼亚州立大学），波兰和乌克兰的研究团队。该项目将在很大程度上建立在乌克兰哈尔科夫数学学院RMT的世界级实力之上。该项目将有助于将哈尔科夫的研究生，博士后和研究人员融入国际科学界和研究队伍，并为在STEM领域工作的新一代乌克兰研究人员做好准备。乌克兰学生将被介绍到最先进的应用RMT在DNN。第2部分.受生物神经网络功能的启发，深度神经网络（DNN）在对象，语音和模式识别等广泛的前沿应用中表现出了其高效率。然而，从理论的角度来看，对它们的理解仍然很差。最近的研究表明，基于随机矩阵理论（RMT）的分析可以帮助提高神经网络的收敛性和学习速度，以及提高深度学习中训练算法的准确性和降低计算复杂度。该项目的重点是双重的：（i）进一步开发RMT技术，（ii）使用这些技术开发深度学习中的数值和分析方法。具体而言，随机矩阵合奏中产生的DNN分析的频谱特性进行了研究，分析和数值。特别是，该项目将通过基于RMT的修剪和正则化技术来提高DNN的准确性并降低计算复杂性。为此，该研究将关注为未经训练的DNN寻找最佳权重初始化，确定训练的停止标准，研究非线性对DNN学习速度的影响，以及证明数值和近似方法以及误差估计。该项目汇集了来自数学和物理学各个领域和方法的研究人员。这些领域包括深度学习、机器学习、RMT、凝聚态物理、概率论和分析。该项目将开发RMT和深度学习的分析和数值工具，丰富全球数学界。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This IMPRESS-U project will be jointly supported by NSF, US National Academy of Sciences, and National Science Centre of Poland. The research will be conducted in collaborative partnership that unites the Pennsylvania State University, USA, Institute for Low Temperature Physics and Engineering of the National Academy of Sciences of Ukraine (ILTPE), and the University of Warsaw, Poland. The USA part of this IMPRESS-U project is co-funded by the Office of International Science and Engineering and MPS/DMS Applied Mathematics and Computational Mathematics programs.Part 1.The main ideas of Artificial Intelligence were formulated in 1970s, when complex technological processes in chemical, automobile and other industries became dependent on an enormous number of random and non-random factors. To answer this challenge, self-learning systems (SLS) were proposed. The SLS allow for fine tuning of production lines and adapting to constantly changing conditions (temperature, pressure, etc). One of the most promising modern realizations of SLS is Deep Neural Networks (DNNs) which are currently used everywhere from industry and defense systems up to internet technologies and cell phones. A key component of the DNNs’ learning process is adjusting the parameters of a DNN to increase its performance. These parameters form matrices with typically initial random entries. Hence, the application of Random Matrix Theory (RMT) could greatly benefit the learning process. In this project, it is expected to establish criteria for optimal learning via developing RMT tools. As a result, it will be possible to speed up the learning process, increase the DNN accuracy, reduce complexity, and avoid over-training.The project engages research teams from the US (Penn State University), Poland, and Ukraine. The project will build to a large extent upon the world-class strength in RMT of the school of mathematics in Kharkiv, Ukraine. The project will help to integrate graduate students, postdocs, and researchers from Kharkiv into the international scientific community and research workforce, and to prepare a new generation of Ukrainian researchers working in STEM fields. The Ukrainian students will be introduced to the state-of-the-art applications of RMT in DNNs.Part 2.Inspired by the function of biological neural networks, Deep Neural Networks (DNNs) have demonstrated their high effectiveness in a wide range of cutting-edge applications such as object, speech, and pattern recognition. However, they are still poorly understood from a theoretical point of view. Recent studies have shown that analysis based on Random Matrix Theory (RMT) can help to improve the convergence and learning speed of neural networks as well as improve accuracy and reduce the computational complexity of training algorithms in deep learning. The focus of the project is twofold: (i) further development of RMT techniques, (ii) employment of these techniques for developing numerical and analytical methods in deep learning. Specifically, the spectral properties of random matrix ensembles arising in the analysis of DNNs will be studied, both analytically and numerically. In particular, the project will address enhancing accuracy of DNNs and reducing computational complexity via RMT-based pruning and regularization techniques. To this end, the research will be concerned with finding an optimal weight initialization for untrained DNNs, determining stopping criteria for training, studying the nonlinearity’s effect on DNN learning speed, as well as justifying numerical and approximation methods and error estimates.The project brings together researchers from various fields and approaches in mathematics and physics. These fields range from deep learning, machine learning, RMT, and condensed matter physics to probability theory, and analysis. The project will develop analytical and numerical tools in RMT and deep learning and enrich the global mathematical community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.