CAREER: Toward a Comprehensive Generalization Theory for Deep Learning

职业：走向深度学习的综合泛化理论

• 批准号: 2045685
• 负责人: Tengyu Ma
• 金额: $ 55万
• 依托单位: Stanford University
• 依托单位国家: 美国
• 项目类别: Continuing Grant
• 财政年份: 2021
• 资助国家: 美国
• 起止时间: 2021-03-01 至 2026-02-28
• 项目状态: 未结题
• 来源: https://www.nsf.gov/awardsearch/showAward?AWD_ID=2045685&HistoricalAwards=false
• 关键词: CAREER; Toward; Comprehensive; Generalization; Theory

The advancement of deep learning, the technique of training artificial neural networks to make predictions, has led to recent breakthroughs in many areas of artificial intelligence, such as computer vision, natural language understanding, and robotics. A major challenge in deep learning is ensuring accurate predictions on unseen scenarios. This project plans to tackle this challenge via theoretical analysis and its empirical evaluation. The project aims to contribute to the fundamental understanding of deep learning and inform the practical advancement of deep learning, improving its reliability, efficiency, and risk management in data-hungry and risk-sensitive applications. An education plan is integrated into this project --- the investigator will develop new courses, mentor students, organize workshops, and work with high-school teachers on developing high-school AI courses.The project aims to build a comprehensive generalization theory for deep neural networks, which covers the technical question of implicit regularization effect and the broad concepts of out-of-domain generalization and the estimation of generalization errors. This project has three major components. The first thrust is to characterize the optimizers’ implicit regularization effect for complex models. Leveraging the theoretical insights, the investigator will make implicit regularization more explicit, stronger, and customizable to datasets to improve generalization. The second thrust is to theoretically study the out-of-domain generalization in settings with an increasing level of differences between the training and test environments by a growing level of exploitation of unlabeled data and their properties. Finally, the PI will study estimating the generalization errors, which is crucial for quantifying the risk before deploying machine learning models in risk-sensitive applications such as healthcare.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.

深度学习（训练人工神经网络进行预测的技术）的进步导致了人工智能许多领域的最新突破，例如计算机视觉、自然语言理解和机器人技术。深度学习的一个主要挑战是确保对未见过的场景进行准确预测。该项目计划通过理论分析和实证评估来应对这一挑战。该项目旨在促进对深度学习的基本理解，并为深度学习的实际进步提供信息，提高其在数据密集型和风险敏感型应用中的可靠性、效率和风险管理。 该项目融入了教育计划——研究者将开发新课程、指导学生、组织研讨会，并与高中教师合作开发高中人工智能课程。该项目旨在建立深度神经网络的综合泛化理论，涵盖隐式正则化效应的技术问题以及域外泛化和泛化误差估计的广泛概念。该项目由三个主要部分组成。第一个重点是表征优化器对复杂模型的隐式正则化效果。利用理论见解，研究人员将使隐式正则化更加明确、更强，并且可针对数据集进行定制，以提高泛化能力。第二个重点是从理论上研究由于对未标记数据及其属性的利用程度不断提高而导致训练和测试环境之间的差异越来越大的情况下的域外泛化。最后，PI 将研究估计泛化误差，这对于在医疗保健等风险敏感应用中部署机器学习模型之前量化风险至关重要。该奖项反映了 NSF 的法定使命，并通过使用基金会的智力优点和更广泛的影响审查标准进行评估，被认为值得支持。