Collaborative Research: Scalable Linear Algebra and Neural Network Theory

点击翻译按钮获取中文摘要

These projects will use randomized numerical linear algebra building blocks to develop improved methods in stochastic optimization theory and statistical/machine learning theory. The motivation is that, while machine learning and deep learning methodology has transformed certain applications, such as computer vision and natural language processing, its promised impact on many other areas has yet to be seen. The reason for this is the flip side of why it has been successful where it has. In the applications where it has had the most remarkable successes, people have adopted the following strategy: get large quantities of data; train a neural network model using stochastic first order methods; and implement and apply the model in a user-facing industrial application. There are many well-known limitations with this general approach, ranging from the need for large quantities of data and daunting compute resources to interpretability and robustness issues. These limitations are particularly apparent when using neural networks for problems such as high-performance computing, fluid mechanics/dynamics, temporal supply chain forecasting problems, biotechnology, etc., where interpretability is paramount. This work aims to address central technical issues underlying this approach, namely: while linear algebraic techniques are central to the design and use of modern neural network models, current methodology uses linear algebra in relatively superficial ways. If we have stronger control over the linear algebraic methods, the community will have a more practical theory to guide neural network use in a broad range of applications beyond computer vision and natural language processing. These methods will enable qualitatively more refined scalable implementations and applications of neural network models in a range of scientific and engineering domains. Broader impacts of these projects include mentoring of grant-supported graduate students and postdoctoral researchers.Technically, the work will focus on three general directions: optimization theory, including convex optimization based neural network and going beyond optimization; scalable linear algebra theory, including randomized linear algebra for neural networks, and sparse randomized linear algebra; and statistics and machine learning theory, including implicit regularization, and learning with limited non-iid data. More broadly, the goal is to provide a basis for practical theory that can guide practice, in a manner analogous to how linear algebraic and functional analytic methods underlie practical and useful theory in a broad range of scientific/engineering applications. We expect that such a challenging task is possible since many of the recent developments in machine learning theory and neural network practice have parallels in scientific computing, where there is a long history of what may be called scalable linear algebra for physical/engineering theory. Many of the methods to be developed may be viewed as bridging the interdisciplinary gap between these old ideas and the new challenges we face; and principal investigators have a history of developing interdisciplinary classes, summer schools, workshops related to the topics of the proposed work, and they will continue to do so.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.