Collaborative Research: Scalable Linear Algebra and Neural Network Theory

这些项目将使用随机数值线性代数构建块来开发随机优化理论和统计/机器学习理论的改进方法。动机是，虽然机器学习和深度学习方法已经改变了某些应用，如计算机视觉和自然语言处理，但它对许多其他领域的预期影响尚未显现。其原因是它在某些地方取得成功的另一面。在它取得最显著成功的应用中，人们采用了以下策略：获取大量数据；利用随机一阶方法训练神经网络模型；并在面向用户的工业应用中实现和应用该模型。这种通用方法有许多众所周知的限制，从需要大量数据和令人生畏的计算资源到可解释性和健壮性问题。当使用神经网络解决高性能计算、流体力学/动力学、时间供应链预测问题、生物技术等问题时，这些局限性尤为明显，因为这些问题的可解释性至关重要。这项工作旨在解决这种方法背后的核心技术问题，即：虽然线性代数技术是现代神经网络模型设计和使用的核心，但目前的方法以相对肤浅的方式使用线性代数。如果我们对线性代数方法有更强的控制，社区将有一个更实用的理论来指导神经网络在计算机视觉和自然语言处理之外的广泛应用。这些方法将使神经网络模型在一系列科学和工程领域的实现和应用在质量上更加精细。这些项目更广泛的影响包括指导资助的研究生和博士后研究人员。在技术上，工作将集中在三个方向：优化理论，包括基于神经网络的凸优化和超越优化；可扩展线性代数理论，包括神经网络的随机线性代数和稀疏随机线性代数；还有统计学和机器学习理论，包括隐式正则化，以及有限非id数据的学习。更广泛地说，目标是为指导实践的实践理论提供基础，类似于线性代数和泛函分析方法如何在广泛的科学/工程应用中奠定实用和有用的理论基础。我们期望这样一个具有挑战性的任务是可能的，因为机器学习理论和神经网络实践的许多最新发展在科学计算中都有相似之处，在科学计算中，物理/工程理论的可扩展线性代数有着悠久的历史。许多有待开发的方法可以被视为弥合这些旧思想与我们面临的新挑战之间的跨学科差距；而且，主要研究人员有开发跨学科课程、暑期学校和与拟议工作主题相关的研讨会的历史，他们将继续这样做。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

These projects will use randomized numerical linear algebra building blocks to develop improved methods in stochastic optimization theory and statistical/machine learning theory. The motivation is that, while machine learning and deep learning methodology has transformed certain applications, such as computer vision and natural language processing, its promised impact on many other areas has yet to be seen. The reason for this is the flip side of why it has been successful where it has. In the applications where it has had the most remarkable successes, people have adopted the following strategy: get large quantities of data; train a neural network model using stochastic first order methods; and implement and apply the model in a user-facing industrial application. There are many well-known limitations with this general approach, ranging from the need for large quantities of data and daunting compute resources to interpretability and robustness issues. These limitations are particularly apparent when using neural networks for problems such as high-performance computing, fluid mechanics/dynamics, temporal supply chain forecasting problems, biotechnology, etc., where interpretability is paramount. This work aims to address central technical issues underlying this approach, namely: while linear algebraic techniques are central to the design and use of modern neural network models, current methodology uses linear algebra in relatively superficial ways. If we have stronger control over the linear algebraic methods, the community will have a more practical theory to guide neural network use in a broad range of applications beyond computer vision and natural language processing. These methods will enable qualitatively more refined scalable implementations and applications of neural network models in a range of scientific and engineering domains. Broader impacts of these projects include mentoring of grant-supported graduate students and postdoctoral researchers.Technically, the work will focus on three general directions: optimization theory, including convex optimization based neural network and going beyond optimization; scalable linear algebra theory, including randomized linear algebra for neural networks, and sparse randomized linear algebra; and statistics and machine learning theory, including implicit regularization, and learning with limited non-iid data. More broadly, the goal is to provide a basis for practical theory that can guide practice, in a manner analogous to how linear algebraic and functional analytic methods underlie practical and useful theory in a broad range of scientific/engineering applications. We expect that such a challenging task is possible since many of the recent developments in machine learning theory and neural network practice have parallels in scientific computing, where there is a long history of what may be called scalable linear algebra for physical/engineering theory. Many of the methods to be developed may be viewed as bridging the interdisciplinary gap between these old ideas and the new challenges we face; and principal investigators have a history of developing interdisciplinary classes, summer schools, workshops related to the topics of the proposed work, and they will continue to do so.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.