III: Small: Towards the Foundations of Training Deep Neural Networks: New Theory and Algorithms

在过去的十年里，深度学习取得了巨大的成功。尽管取得了这些经验上的成功，但对深度学习的理论理解仍在很大程度上落后。深度学习的经验成果与传统的优化和机器学习理论之间存在着巨大的差距。该项目旨在通过建立深度学习的理论基础来了解其为什么和如何工作，并利用这一理论来开发新的模型和算法，从而弥合这一差距。该项目的预期成果包括深度学习的新理论和最先进的方法。该项目将推动深度学习的前沿，并培养下一代人工智能研究人员和从业者。将向K-12学校的学生提供研究演示和实验室参观，展示人工智能的广泛应用及其与社会的联系，以激励他们追求STEM学科。本项目包括两个协同研究主题：(1)了解深度学习模型的随机梯度下降等训练算法的优化动态，并推导出依赖于算法的泛化误差界，以评估其泛化性能；(2)开发一套新的更快的深度学习训练算法，以及以泛化误差界为指导的原则性神经结构搜索算法，以设计更好的神经网络模型。为了评估开发的方法，理论分析和广泛的实验评估将在真实世界的基准上进行，包括但不限于图像分类和自然语言处理。在这个项目中开发的开源软件和课程材料将向更广泛的社区公开，以帮助工程师和科学家更好地理解和应用深度学习。该奖项反映了NSF的法定使命，并通过使用基金会的智力优势和更广泛的影响审查标准进行评估，被认为值得支持。

Deep learning has achieved tremendous successes in the past decade. Despite these empirical successes, the theoretical understanding of deep learning is still largely falling behind. There exists a huge gap between the empirical successes of deep learning and conventional optimization and machine learning theories. This project aims to bridge this gap by establishing the theoretical foundations of deep learning to understand why and how it works, and use this theory to develop new models and algorithms. The expected outcome of this project includes new theories and the state-of-the-art approaches for deep learning. The project will push the frontier of deep learning and train next-generation researchers and practitioners in artificial intelligence. Research demonstrations and lab tours will be given to K-12 school students by showing the wide range of applications of AI and their connection to society, to motivate them to pursue a STEM discipline.This project consists of two synergistic research thrusts: (1) understanding the optimization dynamics of training algorithms such as stochastic gradient descent for deep learning models, and deriving algorithm-dependent generalization error bounds to assess their generalization performance; and (2) developing a new suite of faster training algorithms for deep learning, as well as principled neural architecture search algorithms guided by the generalization error bounds to design better neural network models. To evaluate the developed approaches, both theoretical analyses and extensive experimental evaluations will be performed on real-world benchmarks including but not limited to image classification and natural language processing. The open source software and course materials developed in this project will be made publicly available to the broader community, to help engineers and scientists better understand and apply deep learning.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.