Collaborative Research: SCALE MoDL: Adaptivity of Deep Neural Networks

该项目的首要主题是通过“适应性”的透镜系统地扩展对深度神经网络（DNN）如何工作以及它们为什么或何时优于经典方法的理解。自适应性是指算法的特性，它利用输入数据中的有利结构，而不知道这些结构的存在。也就是说，自适应算法是那些不需要调整参数，并且可以自动配置自己以适应每个输入数据的算法。该项目的预期成果包括一个新的理论，该理论解释和量化了流行的DNN模型的自适应性，如多层感知器、自注意机制（即Transformer模型）和元学习。该理论可以大大节省这些模型的统计和计算复杂性，使它们能够应用于资源受限的环境，并具有更环保的能源足迹。本项目亦将为学生及博士后提供机会，探讨与深度学习相关的跨学科研究课题。具体而言，本项目研究（1）DNN在从噪声数据中估计函数时的“局部自适应性”;（2）自注意机制解析结构数据点时的“关系自适应性（例如图像或文本块）;以及（3）多任务和元学习算法的“任务适应性”，这些算法学习跨多个任务共享信息。该研究涵盖了一些最流行的DNN模型。从技术上讲，该项目利用了数学的多个分支（如函数类，非参数统计，统计学习理论，优化和压缩感知），并涉及近似理论理解，算法见解和DNN统计理论的创新。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The overarching theme of the project is to systematically expand understanding of how deep neural networks (DNNs) work and why or when they are better than classical methods through the lens of "adaptivity." Adaptivity refers to the properties of an algorithm that take advantage of favorable structures in the input data without knowing that these structures exist. That is, adaptive algorithms are those that are free of tuning parameters and could automatically configure themselves to adapt to each input data. The anticipated outcome of the project includes a new theory that explains and quantifies the adaptivity of popular DNN models such as multi-layer perceptrons, self-attention mechanisms (namely, transformer models), and meta-learning. The theory could result in substantial savings in the statistical and computational complexity of these models, allowing them to be applied in resource-constrained settings and to have more environmentally friendly energy footprint. This project will also provide opportunities for students and postdocs to explore interdisciplinary research topics related to deep learning.Specifically, this project investigates (1) the "local adaptivity" of DNNs in estimating functions from noisy data; (2) the "relational adaptivity" of self-attention mechanism that parses a structure data point (such as an image or a chunk of text); and (3) the "task adaptivity" of multi-task and meta-learning algorithms that learn to share information across multiple tasks. The research covers some of the most popular DNN models. Technically the project leverages multiple branches of mathematics (such as function classes, nonparametric statistics, statistical learning theory, optimization, and compressed sensing) and involves innovations in the approximation-theoretic understanding, algorithmic insights, and statistical theory of DNNs. The new analytical tools to be developed are also of independent interest to the broader machine learning theory community.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.