CAREER: Toward a Comprehensive Generalization Theory for Deep Learning

深度学习的进步，即训练人工神经网络进行预测的技术，最近在人工智能的许多领域取得了突破，例如计算机视觉，自然语言理解和机器人技术。深度学习的一个主要挑战是确保对未知场景的准确预测。本项目计划通过理论分析和实证评估来应对这一挑战。该项目旨在促进对深度学习的基本理解，并为深度学习的实际发展提供信息，提高其在数据饥渴和风险敏感应用中的可靠性，效率和风险管理。 这个项目中融入了一个教育计划-研究者将开发新课程，指导学生，组织研讨会，并与高中教师合作开发高中AI课程。该项目旨在为深度神经网络构建一个全面的泛化理论，它涵盖了隐式正则化效应的技术问题和广泛的概念，领域泛化和泛化误差估计。该项目有三个主要组成部分。第一个目标是描述优化器对复杂模型的隐式正则化效果。利用理论见解，研究人员将使隐式正则化更明确，更强大，并可定制数据集，以提高泛化能力。第二个重点是通过对未标记数据及其属性的越来越多的利用，在训练和测试环境之间的差异越来越大的情况下，从理论上研究域外泛化。最后，PI将研究泛化误差估计，这对于在医疗保健等风险敏感应用中部署机器学习模型之前量化风险至关重要。该奖项反映了NSF的法定使命，并通过使用基金会的知识价值和更广泛的影响审查标准进行评估，被认为值得支持。

The advancement of deep learning, the technique of training artificial neural networks to make predictions, has led to recent breakthroughs in many areas of artificial intelligence, such as computer vision, natural language understanding, and robotics. A major challenge in deep learning is ensuring accurate predictions on unseen scenarios. This project plans to tackle this challenge via theoretical analysis and its empirical evaluation. The project aims to contribute to the fundamental understanding of deep learning and inform the practical advancement of deep learning, improving its reliability, efficiency, and risk management in data-hungry and risk-sensitive applications. An education plan is integrated into this project --- the investigator will develop new courses, mentor students, organize workshops, and work with high-school teachers on developing high-school AI courses.The project aims to build a comprehensive generalization theory for deep neural networks, which covers the technical question of implicit regularization effect and the broad concepts of out-of-domain generalization and the estimation of generalization errors. This project has three major components. The first thrust is to characterize the optimizers’ implicit regularization effect for complex models. Leveraging the theoretical insights, the investigator will make implicit regularization more explicit, stronger, and customizable to datasets to improve generalization. The second thrust is to theoretically study the out-of-domain generalization in settings with an increasing level of differences between the training and test environments by a growing level of exploitation of unlabeled data and their properties. Finally, the PI will study estimating the generalization errors, which is crucial for quantifying the risk before deploying machine learning models in risk-sensitive applications such as healthcare.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.