CAREER: Toward a Comprehensive Generalization Theory for Deep Learning

深度学习是一种训练人工神经网络进行预测的技术，它的进步最近导致了人工智能的许多领域的突破，如计算机视觉、自然语言理解和机器人技术。深度学习的一个主要挑战是确保对未知场景的准确预测。本项目计划通过理论分析和实证评估来应对这一挑战。该项目旨在促进对深度学习的基本了解，并为深度学习的实际进展提供信息，提高其可靠性、效率和在数据匮乏和对风险敏感的应用程序中的风险管理。该项目整合了一个教育计划-调查者将开发新课程，指导学生，组织研讨会，并与高中教师合作开发高中人工智能课程。该项目旨在建立一个全面的深层神经网络泛化理论，其中包括隐含正则化效应的技术问题，以及域外泛化和泛化误差估计的广泛概念。这个项目有三个主要组成部分。第一个重点是刻画优化器对复杂模型的隐式正则化效应。利用这些理论见解，研究人员将使隐式正则化对数据集更加明确、更强大和可定制，以提高泛化能力。第二个重点是通过对未标记数据及其属性的开发，从理论上研究训练环境和测试环境之间差异越来越大的环境下的域外泛化。最后，PI将研究估计泛化误差，这对于在医疗保健等风险敏感应用中部署机器学习模型之前量化风险至关重要。这一奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

The advancement of deep learning, the technique of training artificial neural networks to make predictions, has led to recent breakthroughs in many areas of artificial intelligence, such as computer vision, natural language understanding, and robotics. A major challenge in deep learning is ensuring accurate predictions on unseen scenarios. This project plans to tackle this challenge via theoretical analysis and its empirical evaluation. The project aims to contribute to the fundamental understanding of deep learning and inform the practical advancement of deep learning, improving its reliability, efficiency, and risk management in data-hungry and risk-sensitive applications. An education plan is integrated into this project --- the investigator will develop new courses, mentor students, organize workshops, and work with high-school teachers on developing high-school AI courses.The project aims to build a comprehensive generalization theory for deep neural networks, which covers the technical question of implicit regularization effect and the broad concepts of out-of-domain generalization and the estimation of generalization errors. This project has three major components. The first thrust is to characterize the optimizers’ implicit regularization effect for complex models. Leveraging the theoretical insights, the investigator will make implicit regularization more explicit, stronger, and customizable to datasets to improve generalization. The second thrust is to theoretically study the out-of-domain generalization in settings with an increasing level of differences between the training and test environments by a growing level of exploitation of unlabeled data and their properties. Finally, the PI will study estimating the generalization errors, which is crucial for quantifying the risk before deploying machine learning models in risk-sensitive applications such as healthcare.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.