EAGER: IMPRESS-U: Random Matrix Theory and its Applications to Deep Learning

EAGER：IMPRESS-U：随机矩阵理论及其在深度学习中的应用

• 批准号: 2401227
• 负责人: Leonid Berlyand
• 金额: $ 30万
• 依托单位: Pennsylvania State Univ University Park
• 依托单位国家: 美国
• 项目类别: Standard Grant
• 财政年份: 2024
• 资助国家: 美国
• 起止时间: 2024-01-01 至 2025-12-31
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2401227&HistoricalAwards=false
• 关键词: EAGER; IMPRESS; Random; Matrix; Theory

This IMPRESS-U project will be jointly supported by NSF, US National Academy of Sciences, and National Science Centre of Poland. The research will be conducted in collaborative partnership that unites the Pennsylvania State University, USA, Institute for Low Temperature Physics and Engineering of the National Academy of Sciences of Ukraine (ILTPE), and the University of Warsaw, Poland. The USA part of this IMPRESS-U project is co-funded by the Office of International Science and Engineering and MPS/DMS Applied Mathematics and Computational Mathematics programs.Part 1.The main ideas of Artificial Intelligence were formulated in 1970s, when complex technological processes in chemical, automobile and other industries became dependent on an enormous number of random and non-random factors. To answer this challenge, self-learning systems (SLS) were proposed. The SLS allow for fine tuning of production lines and adapting to constantly changing conditions (temperature, pressure, etc). One of the most promising modern realizations of SLS is Deep Neural Networks (DNNs) which are currently used everywhere from industry and defense systems up to internet technologies and cell phones. A key component of the DNNs’ learning process is adjusting the parameters of a DNN to increase its performance. These parameters form matrices with typically initial random entries. Hence, the application of Random Matrix Theory (RMT) could greatly benefit the learning process. In this project, it is expected to establish criteria for optimal learning via developing RMT tools. As a result, it will be possible to speed up the learning process, increase the DNN accuracy, reduce complexity, and avoid over-training.The project engages research teams from the US (Penn State University), Poland, and Ukraine. The project will build to a large extent upon the world-class strength in RMT of the school of mathematics in Kharkiv, Ukraine. The project will help to integrate graduate students, postdocs, and researchers from Kharkiv into the international scientific community and research workforce, and to prepare a new generation of Ukrainian researchers working in STEM fields. The Ukrainian students will be introduced to the state-of-the-art applications of RMT in DNNs.Part 2.Inspired by the function of biological neural networks, Deep Neural Networks (DNNs) have demonstrated their high effectiveness in a wide range of cutting-edge applications such as object, speech, and pattern recognition. However, they are still poorly understood from a theoretical point of view. Recent studies have shown that analysis based on Random Matrix Theory (RMT) can help to improve the convergence and learning speed of neural networks as well as improve accuracy and reduce the computational complexity of training algorithms in deep learning. The focus of the project is twofold: (i) further development of RMT techniques, (ii) employment of these techniques for developing numerical and analytical methods in deep learning. Specifically, the spectral properties of random matrix ensembles arising in the analysis of DNNs will be studied, both analytically and numerically. In particular, the project will address enhancing accuracy of DNNs and reducing computational complexity via RMT-based pruning and regularization techniques. To this end, the research will be concerned with finding an optimal weight initialization for untrained DNNs, determining stopping criteria for training, studying the nonlinearity’s effect on DNN learning speed, as well as justifying numerical and approximation methods and error estimates.The project brings together researchers from various fields and approaches in mathematics and physics. These fields range from deep learning, machine learning, RMT, and condensed matter physics to probability theory, and analysis. The project will develop analytical and numerical tools in RMT and deep learning and enrich the global mathematical community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

这个印象深刻的项目将由NSF，美国国家科学院和波兰国家科学中心共同支持。这项研究将通过合作伙伴关系进行，该合作伙伴关系将宾夕法尼亚州立大学，美国国家科学院低温物理与工程研究所（ILTPE）和波兰大学华沙大学进行。该印象深度项目的美国部分由国际科学和工程办公室以及MPS/DMS应用数学和计算数学计划共同资助。第1部分。第1部分。当时，人工智能的主要思想是在1970年代提出的，当时化学，汽车和其他行业的复杂技术过程变得依赖于随机数量和非瑞om的随机数量。为了应对这一挑战，提出了自学系统（SLS）。 SLS允许对生产线进行微调，并适应不断变化的条件（温度，压力等）。深度神经网络（DNN）最有希望的现代实现之一是，目前从行业和国防系统到互联网技术和手机的各地使用。 DNNS学习过程的关键组成部分是调整DNN的参数以提高其性能。这些参数形成通常具有初始随机条目的物品。因此，随机矩阵理论（RMT）的应用可以极大地使学习过程受益。在这个项目中，预计将通过开发RMT工具建立最佳学习的标准。结果，将有可能加快学习过程，提高DNN的准确性，降低复杂性并避免过度训练。该项目与美国（宾夕法尼亚州立大学），波兰和乌克兰的研究团队互动。该项目将在很大程度上建立在乌克兰哈尔基夫数学学院的世界一流的力量上。该项目将有助于将Kharkiv的研究生，博士后和研究人员整合到国际科学界和研究人员，并准备新一代在STEM领域工作的乌克兰研究人员。深层神经网络（DNNS）将介绍RMT的最新应用。但是，从理论的角度来看，它们仍然对它们的理解很差。最近的研究表明，基于随机矩阵理论（RMT）的分析可以帮助提高神经网络的收敛性和学习速度，并提高准确性并降低深度学习中训练算法的计算复杂性。该项目的重点是双重的：（i）进一步开发RMT技术，（ii）这些技术在深度学习中开发数值和分析方法。具体而言，将在分析和数值上研究在DNN分析中产生的随机矩阵集合的光谱特性。特别是，该项目将通过基于RMT的修剪和调节技术来提高DNN的准确性，并降低计算复杂性。为此，该研究将关注为未经培训的DNN找到最佳的重量初始化，从而确定停止培训的标准，研究非线性对DNN学习速度的影响，以及数字和近似方法以及错误估计的合理性。这些领域的范围从深度学习，机器学习，RMT和凝结物理学到概率理论和分析。该项目将在RMT和深度学习中开发分析和数值工具，并丰富全球数学社区。该奖项反映了NSF的法定任务，并使用基金会的知识分子优点和更广泛的影响评估标准，被视为通过评估来获得支持。