Collaborative Research: SCALE MoDL: Adaptivity of Deep Neural Networks

该项目的首要主题是通过“适应性”的视角，系统地扩展对深度神经网络（dnn）如何工作以及它们为什么或何时优于经典方法的理解。自适应性是指算法在不知道这些结构存在的情况下利用输入数据中的有利结构的特性。也就是说，自适应算法是那些不需要调优参数，可以自动配置自己以适应每个输入数据的算法。该项目的预期结果包括一个新的理论，解释和量化流行的深度神经网络模型的适应性，如多层感知器、自注意机制（即变压器模型）和元学习。该理论可以大大节省这些模型的统计和计算复杂性，使它们能够应用于资源有限的环境中，并具有更环保的能源足迹。该项目还将为学生和博士后提供探索与深度学习相关的跨学科研究课题的机会。具体来说，本项目研究了(1)dnn在从噪声数据估计函数时的“局部适应性”；(2)解析结构数据点（如图像或文本块）的自注意机制的“关系自适应”；(3)学习跨多个任务共享信息的多任务和元学习算法的“任务适应性”。这项研究涵盖了一些最流行的深度神经网络模型。从技术上讲，该项目利用了数学的多个分支（如函数类、非参数统计、统计学习理论、优化和压缩感知），并涉及对深度神经网络的近似理论理解、算法见解和统计理论的创新。即将开发的新分析工具也对更广泛的机器学习理论界有独立的兴趣。该奖项反映了美国国家科学基金会的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The overarching theme of the project is to systematically expand understanding of how deep neural networks (DNNs) work and why or when they are better than classical methods through the lens of "adaptivity." Adaptivity refers to the properties of an algorithm that take advantage of favorable structures in the input data without knowing that these structures exist. That is, adaptive algorithms are those that are free of tuning parameters and could automatically configure themselves to adapt to each input data. The anticipated outcome of the project includes a new theory that explains and quantifies the adaptivity of popular DNN models such as multi-layer perceptrons, self-attention mechanisms (namely, transformer models), and meta-learning. The theory could result in substantial savings in the statistical and computational complexity of these models, allowing them to be applied in resource-constrained settings and to have more environmentally friendly energy footprint. This project will also provide opportunities for students and postdocs to explore interdisciplinary research topics related to deep learning.Specifically, this project investigates (1) the "local adaptivity" of DNNs in estimating functions from noisy data; (2) the "relational adaptivity" of self-attention mechanism that parses a structure data point (such as an image or a chunk of text); and (3) the "task adaptivity" of multi-task and meta-learning algorithms that learn to share information across multiple tasks. The research covers some of the most popular DNN models. Technically the project leverages multiple branches of mathematics (such as function classes, nonparametric statistics, statistical learning theory, optimization, and compressed sensing) and involves innovations in the approximation-theoretic understanding, algorithmic insights, and statistical theory of DNNs. The new analytical tools to be developed are also of independent interest to the broader machine learning theory community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.