CAREER: Toward a Foundation of Over-Parameterization

该奖项全部或部分由2021年美国救援计划法案（公法117-2）资助。过度参数化模型是一种现代机器学习预测技术，正在彻底改变应用领域，包括计算机视觉，自然语言处理和机器人技术。然而，这些强大的模型还没有成为许多数据驱动领域的主要方法，因为它们非常需要资源：最近的模型需要花费数百万美元来训练。该项目将通过彻底表征过参数化的理论特性来应对这一挑战。基于这一坚实的基础，研究人员将设计资源高效的方法，使更广泛的受众能够使用现代机器学习技术。教育计划被整合到这个项目中-研究人员将指导学生，开发新课程，组织研讨会，并为高中机器学习课程开发课程材料。该项目有三个主要组成部分。第一个逆冲断层的特征是宽度的过度参数化。其宏伟目标是开发一个统一的理论框架，在此基础上，研究人员将设计新的宽度收缩方法，以减少过度参数化的模型资源需求。第二个重点将发展的原则，从深度的过度参数化的加速和正规化的影响。洞察力将帮助我们设计更显式的优化器和正则化器来压缩模型，使浅层神经网络可以实现与深层神经网络相似的性能。第三个目标是确定预训练的过度参数化模型可以学习表示的条件，以提高下游任务中的样本效率。然后，研究人员将设计在预训练中选择一小部分数据的方法，而不会降低性能，减少计算资源需求。除了理论开发，该项目还旨在实现所有作为开源软件开发的算法，在标准基准上对其进行评估，并将其部署到交通领域的实际应用中。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

This award is funded in whole or in part under the American Rescue Plan Act of 2021 (Public Law 117-2).Over-parameterized models, a modern machine learning technique to make predictions, are revolutionizing application domains, including computer vision, natural language processing, and robotics. However, these powerful models have not yet become the predominant method in many data-driven domains because they are extremely resource-hungry: recent models cost millions of dollars to train. This project will tackle this challenge by thoroughly characterizing the theoretical properties of over-parameterization. Based on this solid foundation, the investigator will design resource-efficient methods to make modern machine learning technologies accessible to a broader audience. An education plan is integrated into this project --- the investigator will mentor students, develop new courses, organize workshops, and develop course materials for a high school machine learning curriculum. This project has three major components. The first thrust characterizes over-parameterization from width. The grand goal is to develop a unified theoretical framework, based on which the investigator will design new width shrinkage methods to reduce over-parameterized model resource requirements. The second thrust will develop principles on the acceleration and regularization effects of over-parameterization from depth. Insights will help us design more explicit optimizers and regularizers to compress the model so shallow neural networks can achieve performance similar to deep neural networks ones. The third thrust will identify conditions under which pre-trained over-parameterized models can learn representations that improve sample efficiency in downstream tasks. The investigator will then design methods that select a small subset of data in pre-training without degrading performance, reducing computational resource requirements. In addition to theoretical developments, the project also aims to implement all algorithms developed as open-source software, evaluate them on standard benchmarks and deploy them on real-world applications in the transportation domain.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.