CAREER: Toward a Foundation of Over-Parameterization

该奖项的全部或部分资金来自2021年美国救援计划法案(公法117-2)。过度参数模型是一种做出预测的现代机器学习技术，正在给包括计算机视觉、自然语言处理和机器人在内的应用领域带来革命性的变化。然而，这些强大的模型还没有成为许多数据驱动领域的主要方法，因为它们极其耗费资源：最近的模型花费了数百万美元进行培训。这个项目将通过彻底描述过度参数化的理论特性来解决这一挑战。基于这一坚实的基础，研究人员将设计资源高效的方法，使更广泛的受众能够接触到现代机器学习技术。教育计划被整合到这个项目中-调查员将指导学生，开发新课程，组织研讨会，并为高中机器学习课程开发课程材料。这个项目有三个主要组成部分。第一个推力从宽度上表现出过度参数化的特征。总的目标是建立一个统一的理论框架，在此基础上，研究人员将设计新的宽度收缩方法，以减少过度参数化的模型资源需求。第二个推力将从深度发展关于过度参数化的加速和正则化效应的原理。洞察力将帮助我们设计更显式的优化器和正则化器来压缩模型，这样浅层神经网络就可以达到与深层神经网络相似的性能。第三个推力将确定预先训练的过度参数化模型可以学习在下游任务中提高样本效率的表示法的条件。然后，研究人员将设计方法，在不降低性能的情况下，在预训练中选择一小部分数据，减少计算资源需求。除了理论发展，该项目还旨在实现所有作为开源软件开发的算法，在标准基准上对它们进行评估，并将它们部署在交通领域的现实世界应用程序中。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Over-parameterized models, a modern machine learning technique to make predictions, are revolutionizing application domains, including computer vision, natural language processing, and robotics. However, these powerful models have not yet become the predominant method in many data-driven domains because they are extremely resource-hungry: recent models cost millions of dollars to train. This project will tackle this challenge by thoroughly characterizing the theoretical properties of over-parameterization. Based on this solid foundation, the investigator will design resource-efficient methods to make modern machine learning technologies accessible to a broader audience. An education plan is integrated into this project --- the investigator will mentor students, develop new courses, organize workshops, and develop course materials for a high school machine learning curriculum. This project has three major components. The first thrust characterizes over-parameterization from width. The grand goal is to develop a unified theoretical framework, based on which the investigator will design new width shrinkage methods to reduce over-parameterized model resource requirements. The second thrust will develop principles on the acceleration and regularization effects of over-parameterization from depth. Insights will help us design more explicit optimizers and regularizers to compress the model so shallow neural networks can achieve performance similar to deep neural networks ones. The third thrust will identify conditions under which pre-trained over-parameterized models can learn representations that improve sample efficiency in downstream tasks. The investigator will then design methods that select a small subset of data in pre-training without degrading performance, reducing computational resource requirements. In addition to theoretical developments, the project also aims to implement all algorithms developed as open-source software, evaluate them on standard benchmarks and deploy them on real-world applications in the transportation domain.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.