CAREER: Neural Networks in the Practical Regime

该奖项的全部或部分资金来自《2021年美国救援计划法案》(公法117-2)。深度学习是现代人工智能中的主要方法，它使用多层人工神经网络从数据中推断复杂的关系。尽管深度学习在科学和工业应用中迅速被采用，但解释其成功和防范其局限性的方法的理论基础尚未建立。该项目解决了当前深度学习理论中的两个关键空白，即处理特定类型数据的效果和人工神经网络在实际环境中的行为。这项研究对于开发更有效、更可靠、更安全、更广泛适用的深度学习方法至关重要。一些将直接受益的领域包括加快生产管道，提高能源效率，以及在越来越多依赖深度学习的关键技术中改进数据处理。该项目涉及重大的教育、社区建设和外展活动。这项研究将直接融入跨学科课程，并将为研究生产生研究项目主题，为本科生产生顶峰项目。该项目旨在通过为本科生、研究生和博士后提供长期研究和职业指导，在不同职业阶段与专家和爱好者举行公开研讨会和讨论会议，提供实习机会和专门的培训课程，培养多样化的STEM员工队伍。人工神经网络为学习任务的特定候选解决方案集提供特定的参数化，参数优化程序在可能的解决方案空间中引入特定的偏好。本项目力求在现有数学理论没有充分涵盖的实际感兴趣的情况下，即网络相对于训练数据量具有中等程度的过度参数化的情况下，阐明这些复杂关系。重要的是，它发展了集成和利用训练数据和参数初始化的特性的理论和方法。该研究涉及以下三个目标：(1)适度过参数网络的函数空间描述；(2)目标函数和最优化的数据相关描述；(3)函数空间中梯度下降偏差的显式表示。该研究计划通过整合训练数据的属性并解决相互作用中的优化偏差和模型偏差来推进最新技术，这些都是现有方法范围之外的挑战。该项目的基础是融合了应用数学和深度学习的前期工作，特别是连接参数空间、函数空间和数据空间的几何技术，以及基于信息几何、最优传输和代数统计的技术。该项目将进一步发展几何、概率、统计学和机器学习之间的重要联系，并将为跨学科研究和教育提供独特的机会。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2). Deep learning is the predominant approach in modern artificial intelligence that uses multi-layer artificial neural networks to infer complex relations from data. Despite the rapid adoption of deep learning in scientific and industrial applications, a theoretical basis to explain its success and methods to guard against its limits has yet to be established. This project addresses two critical gaps in the current theory of deep learning, namely the effects of working with specific types of data and the behavior of artificial neural networks in practical settings. This research is crucial to developing deep learning methods that are more efficient, reliable, safe, and broadly applicable. Some areas that would benefit directly are accelerating production pipelines, improving energy efficiency, and improving data handling in the increasing number of critical technologies that rely on deep learning. This project involves significant educational, community building, and outreach activities. The research will be directly integrated into interdisciplinary curricula and will generate research project topics for graduate students and capstone projects for undergraduate students. The project aims to prepare a diverse STEM workforce through long-term research and career mentoring for undergraduate and graduate students and postdocs, open seminars and discussion sessions with experts and enthusiasts at different career stages, internship opportunities, and dedicated training sessions.Artificial neural networks provide specific parametrizations to specific sets of candidate solutions to learning tasks, and parameter optimization procedures introduce specific preferences in the space of possible solutions. This project seeks to illuminate these complex relations in cases of practical interest that are not sufficiently well covered by existing mathematical theory, namely, where the networks have a moderate level of overparametrization in relation to the amount of training data. Importantly, it develops theories and methods that integrate and exploit the properties of the training data and parameter initialization. The research concerns the following three aims: (1) the function space description of moderately overparametrized networks, (2) the data-dependent description of the objective function and optimization, and (3) the explicit form of the bias of gradient descent in function space. The research program advances the state of the art by integrating the properties of the training data and addressing the optimization bias and model bias in interplay, which are challenges beyond the scope of existing methods. The project builds on preliminary work that blends applied mathematics and deep learning, in particular techniques connecting the geometry of parameter space, function space, and data space, as well as techniques based on information geometry, optimal transport, and algebraic statistics. The project will further develop important connections between geometry, probability, statistics, and machine learning and will offer unique opportunities for interdisciplinary research and education.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.